Feb 3From weeks to seconds: building “mega-assets” to uncover joyful moments!Swati Sapar, Blaid Burgess, Jonathan Solorzano-Hamilton, Vinod Uddaraju Netflix believes that great stories can come from anywhere and be loved everywhere. Creating and publishing the content to support this strategy poses a tremendous challenge: with thousands of titles available to over two hundred million members in dozens of countries, how…Promotional5 min read
Published in Netflix TechBlog·Feb 1Demystifying Interviewing for Backend Engineers @ NetflixBy Karen Casella, Director of Engineering, Access & Identity Management Have you ever experienced one of the following scenarios while looking for your next role? You study and practice coding interview problems for hours/days/weeks/months, only to be asked to merge two sorted lists. You apply for multiple roles at the…Recruiting7 min read
Published in Netflix TechBlog·Jan 25Netflix: A Culture of LearningMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Colin McFarland, Mihir Tendulkar, and Travis Brooks This is the last post in an overview series on experimentation at Netflix. Need to catch up? Earlier posts covered the basics of A/B tests (Part 1 and Part 2 ), core statistical concepts…Ab Testing13 min read
Published in Netflix TechBlog·Jan 24Fixing Performance Regressions Before they HappenAngus Croll Netflix is used by 222 million members and runs on over 1700 device types ranging from state-of-the-art smart TVs to low-cost mobile devices. At Netflix we’re proud of our reliability and we want to keep it that way. To that end, it’s important that we prevent significant performance…Performance12 min read
Published in Netflix TechBlog·Jan 14Auto-Diagnosis and Remediation in Netflix Data PlatformBy Vikram Srivastava and Marcelo Mayworm Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. As our subscribers grow worldwide and Netflix enters the world of gaming, the number of batch workflows and real-time data…Big Data6 min read
Published in Netflix TechBlog·Jan 11Experimentation is a major focus of Data Science across NetflixMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Colin McFarland, Andy Rhines, Sophia Liu, Mihir Tendulkar, Kevin Mercurio, Veronica Hannan, Ting-Po Lee Earlier posts in this series covered the basics of A/B tests (Part 1 and Part 2 ), core statistical concepts (Part 3 and Part 4), and how…Ab Testing18 min read
Published in Netflix TechBlog·Dec 8, 2021Snaring the Bad FolksProject by Netflix’s Cloud Infrastructure Security team (Alex Bainbridge, Mike Grima, Nick Siow) — Cloud security is a hard problem, but an even harder one is cloud security at scale. In recent years we’ve seen several cloud focused data breaches and evidence shows that threat actors are becoming more advanced with their techniques, goals, and tooling. With 2021 set to be a new high…AWS6 min read
Nov 26, 2021Cache warming: Leveraging EBS for moving petabytes of databy Sailesh Mukil, Prudhviraj Karumanchi, Tharanga Gamaethige, Shashi Madappa Introduction EVCache is a distributed in-memory caching solution based on memcached. It is a Tier-0 system at Netflix with its footprint across ~18,000 servers holding ~14 petabytes of data, and still rapidly growing. We had previously given an overview of how we…Cache16 min read
Published in Netflix TechBlog·Nov 15, 2021Building confidence in a decisionMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Michael Lindon, and Colin McFarland This is the fifth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making…Ab Testing9 min read
Published in Netflix TechBlog·Nov 9, 2021Bringing AV1 Streaming to Netflix Members’ TVsby Liwei Guo, Ashwin Kumar Gopi Valliammal, Raymond Tam, Chris Pham, Agata Opalach, Weibo Ni AV1 is the first high-efficiency video codec format with a royalty-free license from Alliance of Open Media (AOMedia), made possible by wide-ranging industry commitment of expertise and resources. Netflix is proud to be a founding…Av 18 min read
Published in Netflix TechBlog·Nov 2, 2021Netflix Video Quality at Scale with Cosmos Microservicesby Christos G. Bampis, Chao Chen, Anush K. Moorthy and Zhi Li Introduction Measuring video quality at scale is an essential component of the Netflix streaming pipeline. Perceptual quality measurements are used to drive video encoding optimizations, perform video codec comparisons, carry out A/B testing and optimize streaming QoE decisions to…Cosmos Microservice7 min read
Published in Netflix TechBlog·Oct 28, 2021Data Engineers of Netflix — Interview with Pallavi PhadnisThis post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Pallavi Phadnis is a Senior Software Engineer on the Product Data Science and Engineering team. In this post, Pallavi talks about her journey to…Data Engineering4 min read
Published in Netflix TechBlog·Oct 28, 2021Open-Sourcing a Monitoring GUI for Metaflow, Netflix’s ML Platformtl;dr Today, we are open-sourcing a long-awaited GUI for Metaflow. The Metaflow GUI allows data scientists to monitor their workflows in real-time, track experiments, and see detailed logs and results for every executed task. The GUI can be extended with plugins, allowing the community to build integrations to other systems…Metaflow8 min read
Published in Netflix TechBlog·Oct 26, 2021Interpreting A/B test results: false negatives and powerMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland This is the fourth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making at Netflix)…Ab Testing9 min read
Published in Netflix TechBlog·Oct 15, 2021Data pipeline asset management with Dataflowby Sam Setegne, Jai Balani, Olek Gorajek Glossary asset — any business logic code in a raw (e.g. SQL) or compiled (e.g. JAR) form to be executed as part of the user defined data pipeline. data pipeline — a set of tasks (or jobs) to be executed in a predefined order…Data Management12 min read
Published in Netflix TechBlog·Oct 15, 2021Revisiting BetterTLS: Certificate Path BuildingBy Ian Haken Last year the AddTrust root certificate expired and lots of clients had a bad time. Some Roku devices weren’t working right, Heroku had problems, and some folks couldn’t even curl. In the aftermath Ryan Sleevi wrote a really great blog post not just about the issue of…Bettertls16 min read
Published in Netflix TechBlog·Oct 12, 2021CAMBI, a banding artifact detectorby Joel Sole, Mariana Afonso, Lukas Krasula, Zhi Li, and Pulkit Tandon — Introducing the banding artifacts detector developed by Netflix aiming at further improving the delivered video quality Banding artifacts can be pretty annoying. But, first of all, you may wonder, what is a banding artifact? Banding artifact? You are at home enjoying a show on your brand-new TV. Great content delivered at excellent…Vmaf8 min read
Published in Netflix TechBlog·Oct 7, 2021Safe Updates of Client Applications at NetflixBy Minal Mishra Quality of a client application is of paramount importance to global digital products, as it is the primary way customers interact with a brand. At Netflix, we have significant investments in ensuring new versions of our applications are well tested. However, Netflix is available for streaming on…Continuous Delivery9 min read
Published in Netflix TechBlog·Oct 7, 2021Interpreting A/B test results: false positives and statistical significanceMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland This is the third post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making at Netflix)…Ab Testing11 min read
Published in Netflix TechBlog·Sep 28, 2021How We Build Micro Frontends With LatticeWritten by Michael Possumato, Nick Tomlin, Jordan Andree, Andrew Shim, and Rahul Pilani. As we continue to grow here at Netflix, the needs of Revenue and Growth Engineering are rapidly evolving; and our tools must also evolve just as rapidly. The Revenue and Growth Tools (RGT) team decided to set…Micro Frontends7 min read
Published in Netflix TechBlog·Sep 24, 2021Netflix Cloud Packaging in the Terabyte EraBy Xiaomei Liu, Rosanna Lee, Cyril Concolato Introduction Behind the scenes of the beloved Netflix streaming service and content, there are many technology innovations in media processing. Packaging has always been an important step in media processing. After content ingestion, inspection and encoding, the packaging step encapsulates encoded video and audio…9 min read
Published in Netflix TechBlog·Sep 22, 2021What is an A/B Test?Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland This is the second post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. See here for Part 1: Decision Making at Netflix. Subsequent posts will go into…Ab Testing9 min read
Published in Netflix TechBlog·Sep 15, 2021Practical API Design at Netflix, Part 2: Protobuf FieldMask for Mutation OperationsBy Ricky Gardiner, Alex Borysov Background In our previous post, we discussed how we utilize FieldMask as a solution when designing our APIs so that consumers can request the data they need when fetched via gRPC. …API5 min read
Published in Netflix TechBlog·Sep 13, 2021The Show Must Go On: Securing Netflix Studios At ScaleWritten by Jose Fernandez, Arthur Gonigberg, Julia Knecht, and Patrick Thomas In 2017, Netflix Studios was hitting an inflection point from a period of merely rapid growth to the sort of explosive growth that throws “how do we scale?” into every conversation. The vision was to create a “Studio in…Netflix11 min read
Published in Netflix TechBlog·Sep 7, 2021Decision Making at NetflixMartin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland This introduction is the first in a multi-part series on how Netflix uses A/B tests to make decisions that continuously improve our products, so we can deliver more joy and satisfaction to our members. Subsequent posts will cover…Ab Testing3 min read
Published in Netflix TechBlog·Sep 3, 2021Practical API Design at Netflix, Part 1: Using Protobuf FieldMaskBy Alex Borysov, Ricky Gardiner Background At Netflix, we heavily use gRPC for the purpose of backend to backend communication. When we process a request it is often beneficial to know which fields the caller is interested in and which ones they ignore. Some response fields can be expensive to compute…API8 min read
Sep 3, 2021Improving Pull Request Confidence for the Netflix TV AppBy: Nadeem Ahmad, Ramya Somaskandan Introduction The Netflix TV app is used across millions of smart TVs, streaming media players, gaming consoles, and set-top boxes worldwide. …Developer Productivity10 min read
Published in Netflix TechBlog·Aug 31, 2021Towards a Reliable Device Management PlatformBy Benson Ma, Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In addition, Netflix continuously works with its…Kafka15 min read
Published in Netflix TechBlog·Jul 26, 2021Data Movement in Netflix Studio via Data MeshBy Andrew Nguonly, Armando Magalhães, Obi-Ike Nwoke, Shervin Afshar, Sreyashi Das, Tongliang Liu, Wei Liu, Yucheng Zeng Background Over the next few years, most content on Netflix will come from Netflix’s own Studio. From the moment a Netflix film or series is pitched and long before it becomes available on Netflix…Data Pipeline12 min read
Published in Netflix TechBlog·Jul 15, 2021Data Engineers of Netflix — Interview with Kevin WylieThis post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Kevin Wylie is a Data Engineer on the Content Data Science and Engineering team. In this post, Kevin talks about his extensive experience in…Data Engineering5 min read
Jul 13, 2021Elasticsearch Indexing Strategy in Asset Management Platform (AMP)By Burak Bacioglu, Meenakshi Jindal Asset Management at Netflix At Netflix, all of our digital media assets (images, videos, text, etc.) are stored in secure storage layers. We built an asset management platform (AMP), codenamed Amsterdam, in order to easily organize and manage the metadata, schema, relations and permissions of these assets. It is…Asset Management10 min read
Published in Netflix TechBlog·Jun 25, 2021Exploring Data @ NetflixBy Gim Mahasintunan on behalf of Data Platform Engineering. Supporting a rapidly growing base of engineers of varied backgrounds using different data stores can be challenging in any organization. Netflix’s internal teams strive to provide leverage by investing in easy-to-use tooling that streamlines the user experience and incorporates best practices. …Cassandra8 min read
Published in Netflix TechBlog·Jun 23, 2021Introducing Netflix Timed Text Authoring LineageA Script Authoring Specification By: Bhanu Srikanth, Andy Swan, Casey Wilms, Patrick Pearson The Art of Dubbing and Subtitling Dubbing and subtitling are inherently creative processes. At Netflix, we strive to make shows as joyful to watch in every language as in the original language, whether a member watches with original or dubbed audio, closed captions…Specifications4 min read
Published in Netflix TechBlog·Jun 7, 2021How Netflix uses eBPF flow logs at scale for network insightBy Alok Tiagi, Hariharan Ananthakrishnan, Ivan Porto Carrero and Keerti Lakshminarayan Netflix has developed a network observability sidecar called Flow Exporter that uses eBPF tracepoints to capture TCP flows at near real time. …Ebpf4 min read
Published in Netflix TechBlog·May 29, 2021My (Seemingly) Random Walk to NetflixPart of our series on who works in Analytics at Netflix — and what the role entails — By Sean Barnes, Studio Production Data Science & Engineering I am going to tell you a story about a person that works for Netflix. That person grew up dreaming of working in the entertainment industry. They attended the University of Southern California, double majored in data science and television &…Netfli8 min read
Published in Netflix TechBlog·May 12, 2021Achieving observability in async workflowsWritten by Colby Callahan, Megha Manohara, and Mike Azar. Managing and operating asynchronous workflows can be difficult without the proper tools and architecture that puts observability, debugging, and tracing at the forefront. Imagine getting paged outside normal work hours — users are having trouble with the application you’re responsible for…Asynchronous6 min read
Published in Netflix TechBlog·May 5, 2021Netflix DriveA file and folder interface for Netflix Cloud Services — Written by Vikram Krishnamurthy, Kishore Kasi, Abhishek Kapatkar, Tejas Chopra, Prudhviraj Karumanchi, Kelsey Francis, Shailesh Birari In this post, we are introducing Netflix Drive, a Cloud drive for media assets and providing a high level overview of some of its features and interfaces. We intend this to be a first…Netflix8 min read
Published in Netflix TechBlog·Apr 27, 2021Data Engineers of Netflix — Interview with Samuel SetegneThis post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Samuel Setegne is a Senior Software Engineer on the Core Data Science and Engineering team. Samuel and his team build tools and frameworks…Data Engineering5 min read
Apr 16, 2021Netflix Hack Day — Winter 2021By: Carenina Garcia Motion, Leslie Posada, and Bethany Brodsky Hack Day at Netflix has been a time for employees across the company to come together, meet, and collaborate on making fun ideas about Netflix come to life. Previously we have wanted to include Netflix offices across the globe, however, being…Hackathons3 min read
Published in Netflix TechBlog·Mar 23, 2021Scaling Revenue & Growth ToolingWritten by Nick Tomlin, Michael Possumato, and Rahul Pilani. This post shares how the Revenue & Growth Tools (RGT) team approaches creating full-stack tools for the teams that are the financial backbone of Netflix. Our primary partners are the teams of Revenue and Growth Engineering (RGE): Growth, Membership, Billing, Payments…Tooling6 min read
Published in Netflix TechBlog·Mar 15, 2021Data Engineers of Netflix — Interview with Dhevi RajendranThis post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Dhevi Rajendran is a Data Engineer on the Growth Data Science and Engineering team. Dhevi joined Netflix in July 2020 and is one…Data Engineering5 min read
Published in Netflix TechBlog·Mar 13, 2021Production Media Management: Transforming Media Workflows by leveraging the CloudWritten by Anton Margoline, Avinash Dathathri, Devang Shah and Murthy Parthasarathi. Credit to Netflix Studio’s Product, Design, Content Hub Engineering teams along with all of the supporting partner and platform teams. In this post, we will share a behind-the-scenes look at how Netflix delivers technology and infrastructure to help production…Media Management9 min read
Published in Netflix TechBlog·Mar 10, 2021ConsoleMe: A Central Control Plane for AWS Permissions and AccessConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel, Patrick Sanders, and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. …AWS9 min read
Mar 9, 2021Words Matter: Testing Copy With ShakespeareBy Tim Brandall, Shawn Xu, Pu Chen and Jen Schaefer It sounds like a dream come true for word-focused product people: a tool that lets Content Designers, Language Managers and others run lighter-weight copy tests, without heavy engineering support. At Netflix, this has been the reality for around a year…Internationalization8 min read
Published in Netflix TechBlog·Mar 8, 2021Remote Workstations for the Discerning ArtistsBy Michelle Brenner Netflix is poised to become the world’s most prolific producer of visual effects and original animated content. To meet that demand, we need to attract the world’s best artistic talent. Artists like to work at places where they can create groundbreaking entertainment instead of worrying about getting…Infrastructure4 min read
Mar 3, 2021Data movement for Google services at NetflixBy Raghuram Onti Srinivasan and Tianlong Chen Netflix Studio is creating many hundreds of titles per year and a lot of this information is in varied data sources. We in the Data Platform organization have partners at Netflix Studio using services like Google Sheets and Google Drive for day to…Data Movement6 min read
Published in Netflix TechBlog·Mar 3, 2021A Day in the Life of an Experimentation and Causal Inference Scientist @ NetflixStephanie Lane, Wenjing Zheng, Mihir Tendulkar Within the rapid expansion of data-related roles in the last decade, the title Data Scientist has emerged as an umbrella term for myriad skills and areas of business focus. What does this title mean within a given company, or even within a given industry…Experimentation10 min read
Published in Netflix TechBlog·Mar 2, 2021The Netflix Cosmos PlatformOrchestrated Functions as a Microservice — by Frank San Miguel on behalf of the Cosmos team Introduction Cosmos is a computing platform that combines the best aspects of microservices with asynchronous workflows and serverless functions. Its sweet spot is applications that involve resource-intensive algorithms coordinated via complex, hierarchical workflows that last anywhere from minutes to years. It…Distributed Computing11 min read
Published in Netflix TechBlog·Feb 26, 2021Packaging award-winning shows with award-winning technologyBy Cyril Concolato Introduction In previous blog posts, our colleagues at Netflix have explained how 4K video streams are optimized, how even legacy video streams are improved and more recently how new audio codecs can provide better aural experiences to our members. In all these cases, prior to being delivered through…Streaming6 min read
Published in Netflix TechBlog·Feb 22, 2021Beyond RESTRapid Development with GraphQL Microservices — by Dane Avilla The entertainment industry has struggled with COVID-19 restrictions impacting productions around the globe. Since early 2020, Netflix has been iteratively developing systems to provide internal stakeholders and business leaders with up-to-date tools and dashboards with the latest information on the pandemic. These software solutions allow executive leadership…Graph QL8 min read
Published in Netflix TechBlog·Feb 16, 2021Building a Rule-Based Platform to Manage Netflix Membership SKUs at ScaleBy Budhaditya Das, Wallace Wang, and Scott Yao At Netflix, we aspire to entertain the world. From mailing DVDs in the US to a global streaming service with over 200 million subscribers across 190 countries, we have come a long way. For the longest time, Netflix had three plans (basic/standard/premium)…Membership Engineering10 min read
Published in Netflix TechBlog·Feb 10, 2021Hawkins: Diving into the Reasoning Behind our Design Systemby Hawkins team member Joshua Godi; with cover art from Martin Bekerman and additional imagery from Wiki Chaves Hawkins may be the name of a fictional town in Indiana, most widely known as the backdrop for one of Netflix’s most popular TV series “Stranger Things,” but the name is so…Design Systems9 min read
Published in Netflix TechBlog·Feb 9, 2021Edge Authentication and Token-Agnostic Identity Propagationby AIM Team Members Karen Casella, Travis Nelson, Sunny Singh; with prior art and contributions by Justin Ryan, Satyajit Thadeshwar As most developers can attest, dealing with security protocols and identity tokens, as well as user and device authentication, can be challenging. Imagine having multiple protocols, multiple tokens, 200M+ users…Microservice Architecture14 min read
Published in Netflix TechBlog·Feb 8, 2021Growth Engineering at Netflix- Creating a Scalable Offers Platformby Eric Eiswerth Background Netflix has been offering streaming video-on-demand (SVOD) for over 10 years. Throughout that time we’ve primarily relied on 3 plans (Basic, Standard, & Premium), combined with the 30-day free trial to drive global customer acquisition. The world has changed a lot in this time. Competition for people’s…Growth9 min read
Published in Netflix TechBlog·Feb 3, 2021Open Sourcing the Netflix Domain Graph Service Framework: GraphQL for Spring BootBy Paul Bakker and Kavitha Srinivasan, Images by David Simmer, Edited by Greg Burrell Netflix has developed a Domain Graph Service (DGS) framework and it is now open source. The DGS framework simplifies the implementation of GraphQL, both for standalone and federated GraphQL services. Our framework is battle-hardened by our…Graph QL7 min read
Published in Netflix TechBlog·Feb 1, 2021Growth Engineering at Netflix — Automated Imagery Generationby Eric Eiswerth Background There’s a good chance you’ve probably visited the Netflix homepage. In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on…Software Development8 min read
Published in Netflix TechBlog·Jan 21, 2021Optimizing the Aural Experience on Android Devices with xHE-AACBy Phill Williams and Vijay Gondi Introduction At Netflix, we are passionate about delivering great audio to our members. We began streaming 5.1 channel surround sound in 2010, Dolby Atmos in 2017, and adaptive bitrate audio in 2019. Continuing in this tradition, we are proud to announce that Netflix now streams…Audio11 min read
Published in Netflix TechBlog·Dec 23, 2020Evolving Container Security With Linux User NamespacesBy Fabio Kung, Sargun Dhillon, Andrew Spyker, Kyle Anderson, Rob Gulewich, Nabil Schear, Andrew Leung, Daniel Muino, and Manas Alekar As previously discussed on the Netflix Tech Blog, Titus is the Netflix container orchestration system. It runs a wide variety of workloads from various parts of the company — everything…Containers12 min read
Published in Netflix TechBlog·Dec 21, 2020Optimizing data warehouse storageBy Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3, and each day we ingest and create additional Petabytes. At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions)…Data Platforms11 min read
Published in Netflix TechBlog·Dec 19, 2020Mythbusting the Analytics JourneyPart of our series on who works in Analytics at Netflix — and what the role entails — by Alex Diamond This Q&A aims to mythbust some common misconceptions about succeeding in analytics at a big tech company. This isn’t your typical recruiting story. I wasn’t actively looking for a new job and Netflix was the only place I applied. I didn’t know anyone who worked there and just submitted my resume through the Jobs page 🤷🏼♀️ . I wasn’t even entirely sure what the…Netflix6 min read
Published in Netflix TechBlog·Dec 17, 2020Netflix at MIT CODE 2020Martin Tingley In November, Netflix was a proud sponsor of the 2020 Conference on Digital Experimentation (CODE), hosted by the MIT Initiative on the Digital Economy. As well as providing sponsorship, Netflix data scientists were active participants, with three contributions. Eskil Forsell and colleagues presented a poster describing Success stories…Experimentation2 min read
Published in Netflix TechBlog·Dec 11, 2020Life of a Netflix Partner Engineer — The case of the extra 40 msBy: John Blair, Netflix Partner Engineering The Netflix application runs on hundreds of smart TVs, streaming sticks and pay TV set top boxes. The role of a Partner Engineer at Netflix is to help device manufacturers launch the Netflix application on their devices. In this article we talk about one…Netflix7 min read
Published in Netflix TechBlog·Dec 11, 2020How Netflix Scales its API with GraphQL Federation (Part 2)In our previous post and QConPlus talk, we discussed GraphQL Federation as a solution for distributing our GraphQL schema and implementation. In this post, we shift our attention to what is needed to run a federated GraphQL platform successfully — from our journey implementing it to lessons learned. Our Journey so Far Over the…Graph QL11 min read
Published in Netflix TechBlog·Dec 10, 2020Supporting content decision makers with machine learningby Melody Dye*, Chaitanya Ekanadham*, Avneesh Saluja*, Ashish Rastogi * contributed equally Netflix is pioneering content creation at an unprecedented scale. Our catalog of thousands of films and series caters to 195M+ members in over 190 countries who span a broad and diverse range of tastes. Content, marketing, and studio…Machine Learning9 min read
Published in Netflix TechBlog·Dec 8, 2020Toward a Better Quality Metric for the Video Communityby Zhi Li, Kyle Swanson, Christos Bampis, Lukáš Krasula and Anne Aaron Over the past few years, we have been striving to make VMAF a more usable tool not just for Netflix, but for the video community at large. This tech blog highlights our recent progress toward this goal. VMAF…Vmaf6 min read
Published in Netflix TechBlog·Nov 23, 2020Simple streaming telemetryIntroducing gnmi-gateway: a modular, distributed, and highly available service for modern network telemetry via OpenConfig and gNMI — By: Colin McIntosh, Michael Costello Netflix runs its own content delivery network, Open Connect, which delivers all streaming traffic to our members. A backbone network underlies a large portion of the CDN, and we also run the high capacity networks that support our studios and corporate offices. …Open Connect9 min read
Published in Netflix TechBlog·Nov 9, 2020How Netflix Scales its API with GraphQL Federation (Part 1)Netflix is known for its loosely coupled and highly scalable microservice architecture. Independent services allow for evolving at different paces and scaling independently. Yet they add complexity for use cases that span multiple services. …Graph QL9 min read
Published in Netflix TechBlog·Nov 2, 2020Keeping Netflix Reliable Using Prioritized Load SheddingHow viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure — By Manuel Correa, Arthur Gonigberg, and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Everyone slows to a crawl, sometimes for a minor issue or sometimes for no reason at all. As engineers at Netflix, we are constantly reevaluating how…Load Shedding8 min read
Published in Netflix TechBlog·Oct 31, 2020A Day in the Life of a Content Analytics EngineerPart of our series on who works in Analytics at Netflix — and what the role entails — by Rocio Ruelas Back when we were all working in offices, my favorite days were Monday, Wednesday, and Friday. Those were the days with the best breakfast items on the cafeteria menu! I started the day by arriving at the LA office right before 8am and finding a parking spot…Netflix6 min read
Published in Netflix TechBlog·Oct 29, 2020Netflix Android and iOS Studio Apps — now powered by Kotlin MultiplatformBy David Henry & Mel Yahya Over the last few years Netflix has been developing a mobile app called Prodicle to innovate in the physical production of TV shows and movies. The world of physical production is fast-paced, and needs vary significantly between the country, region, and even from one…Android4 min read
Published in Netflix TechBlog·Oct 27, 2020Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value StoresBy Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. Usually Data scientists…Data Movement8 min read
Published in Netflix TechBlog·Oct 20, 2020Building Netflix’s Distributed Tracing Infrastructureby Maulik Pandey Our Team — Kevin Lew, Narayanan Arunachalam, Elizabeth Carretto, Dustin Haffner, Andrei Ushakov, Seth Katz, Greg Burrell, Ram Vaithilingam, Mike Smith and Maulik Pandey “@Netflixhelps Why doesn’t Tiger King play on my phone?” — a Netflix member via Twitter This is an example of a question our…Distributed Tracing10 min read
Published in Netflix TechBlog·Sep 18, 2020How Our Paths Brought Us to Data and NetflixPart of our series on who works in Analytics at Netflix — and what the role entails — by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley. [Chris] Julie and I joined the Streaming DSE team at Netflix…Netflix5 min read
Published in Netflix TechBlog·Sep 18, 2020Analytics at Netflix: Who We Are and What We DoAn Introduction to Analytics and Visualization Engineering at Netflix — by Molly Jackman & Meghana Reddy Across nearly every industry, there is recognition that data analytics is key to driving informed business decision-making. But there is far less agreement on what that term “data analytics” actually means — or what to call the people responsible for the work. Even within…Netflix5 min read
Published in Netflix TechBlog·Sep 8, 2020Seamlessly Swapping the API backend of the Netflix Android appHow we migrated our Android endpoints out of a monolith into a new microservice — by Rohan Dhruva, Ed Ballot As Android developers, we usually have the luxury of treating our backends as magic boxes running in the cloud, faithfully returning us JSON. At Netflix, we have adopted the Backend for Frontend (BFF) pattern: instead of having one general purpose “backend API”, we have one…Android13 min read
Published in Netflix TechBlog·Sep 4, 2020HDMI — Scaling Netflix CertificationScott Bolter, Matthew Lehman, Akshay Garg ¹ At Netflix, we take the task of preserving the creative vision of our content all the way to a subscriber TV screen very seriously. This significantly increases the scope of our application integration and certification processes for streaming devices like set-top-boxes (STBs) and…Hdmi9 min read
Published in Netflix TechBlog·Sep 3, 2020Edgar: Solving Mysteries Faster with ObservabilityEdgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. — by Elizabeth Carretto Our Team — Kevin Lew, Maulik Pandey, Narayanan Arunachalam, Dustin Haffner, Andrei Ushakov, Seth Katz, Greg Burrell, Ram Vaithilingam, Mike Smith and Elizabeth Carretto Everyone loves Unsolved Mysteries. There’s always someone who seems like the surefire culprit. There’s a clear motive, the perfect opportunity, and an incriminating…Distributed Tracing14 min read
Published in Netflix TechBlog·Sep 1, 2020Key Challenges with Quasi Experiments at NetflixKamer Toker-Yildiz, Colin McFarland, Julia Glick At Netflix, when we can’t run A/B experiments we run quasi experiments! We run quasi experiments with various objectives such as non-member experiments focusing on acquisition, member experiments focusing on member engagement, or video streaming experiments focusing on content delivery. Consolidating on one methodology…Experimentation6 min read
Published in Netflix TechBlog·Aug 29, 2020Optimized shot-based encodes for 4K: Now streaming!by Aditya Mavlankar, Liwei Guo, Anush Moorthy and Anne Aaron Netflix has an ever-expanding collection of titles which customers can enjoy in 4K resolution with a suitable device and subscription plan. Netflix creates premium bitstreams for those titles in addition to the catalog-wide 8-bit stream profiles¹. …Video Encoding9 min read
Published in Netflix TechBlog·Aug 14, 2020Telltale: Netflix Application Monitoring SimplifiedBy Andrei Ushakov, Seth Katz, Janak Ramachandran, Jeff Butsch, Peter Lau, Ram Vaithilingam, and Greg Burrell Our Telltale Vision An alert fires and you get paged in the middle of the night. A metric crossed a threshold. You’re half awake and wondering, “Is there really a problem or is this just an alert…Netflix8 min read
Published in Netflix TechBlog·Aug 12, 2020Computational Causal Inference at NetflixJeffrey Wong, Colin McFarland Every Netflix data scientist, whether their background is from biology, psychology, physics, economics, math, statistics, or biostatistics, has made meaningful contributions to the way Netflix analyzes causal effects. Scientists from these fields have made many advancements in causal effects research in the past few decades, spanning…Causal Inference4 min read
Published in Netflix TechBlog·Aug 10, 2020Improving our video encodes for legacy devicesby Mariana Afonso, Anush Moorthy, Liwei Guo, Lishan Zhu, Anne Aaron Netflix has been one of the pioneers of streaming video-on-demand content — we announced our intention to stream video over 13 years ago, in January 2007 — and have only increased both our device and content reach since then…Encoding4 min read
Published in Netflix TechBlog·Jul 30, 2020Unbundling Data Science Workflows with Metaflow and AWS Step Functionsby David Berg, Ravi Kiran Chirravuri, Romain Cledat, Jason Ge, Savin Goyal, Ferras Hamad, Ville Tuulos tl;dr Today, we are releasing a new job scheduler integration with AWS Step Functions. …Machine Learning10 min read
Published in Netflix TechBlog·Jul 21, 2020Machine Learning for a Better Developer ExperienceStanislav Kirdey, William High Imagine having to go through 2.5GB (not often, but does happen time to time) log entries from a failed software build — 3 million lines — to search for a bug or a regression that happened on line 1M. It’s probably not even doable manually! …Machine Learning7 min read
Published in Netflix TechBlog·Jul 17, 2020Empowering the Visual Effects Community with the NetFX PlatformBy Jeff Shapiro, Piya Wadia, Andy Jones, Mike Schmueck, Peter Sauvey, Tom Wild, Kyle Spiker At Netflix, we want to entertain our global membership with series and films from around the world. In line with that, we’re excited to announce NetFX, a cloud-based platform that will make it easier for…Visual Effects2 min read
Published in Netflix TechBlog·Jul 9, 2020Byte Down: Making Netflix’s Data Infrastructure Cost-EffectiveBy Torio Risianto, Bhargavi Reddy, Tanvi Sahni, Andrew Park Background on data efficiency At Netflix, we invest heavily in our data infrastructure which is composed of dozens of data platforms, hundreds of data producers and consumers, and petabytes of data. At many other organizations, an effective way to manage data infrastructure costs is to…Data8 min read
Published in Netflix TechBlog·Jun 30, 2020Netflix Studio Engineering OverviewBy Steve Urban, Sridhar Seetharaman, Shilpa Motukuri, Tom Mack, Erik Strauss, Hema Kannan, CJ Barker Netflix is revolutionizing the way a modern studio operates. Our mission in Studio Engineering is to build a unified, global, and digital studio that powers the effective production of amazing content. …Engineering3 min read
Published in Netflix TechBlog·May 27, 2020Keeping Customers Streaming — The Centralized Site Reliability Practice at NetflixBy Hank Jacobs, Senior Site Reliability Engineer on CORE We’re privileged to be in the business of bringing joy to our customers at Netflix. Whether it’s a compelling new series or an innovative product feature, we strive to provide a best-in-class service that people love and can enjoy anytime, anywhere…Site Reliability4 min read
Published in Netflix TechBlog·May 26, 2020How Netflix is able to enrich VPC Flow Logs at Hyper Scale to provide Network InsightBy Hariharan Ananthakrishnan and Angela Ho The Cloud Network Infrastructure that Netflix utilizes today is a large distributed ecosystem that consists of specialized functional tiers and services such as DirectConnect, VPC Peering, Transit Gateways, NAT Gateways, etc. …Netflix5 min read
Published in Netflix TechBlog·Apr 20, 2020How Netflix brings safer and faster streaming experiences to the living room on crowded networks using TLS 1.3By Sekwon Choi At Netflix, we are obsessed with the best streaming experiences. We want playback to start instantly and to never stop unexpectedly in any network environment. We are also committed to protecting users’ privacy and service security without sacrificing any part of the playback experience. To achieve that…Streaming5 min read
Published in Netflix TechBlog·Apr 2, 2020Bringing 4K and HDR to Anime at Netflix with Sol LevanteBy Haruka Miyagawa & Kylee Peña Some might dismiss them as simple cartoons, but anime’s diverse and fantastical stories, vibrant style, and delicate lines are an art form that has evolved and grown in popularity, variety and sophistication over the last fifty years. …Anime15 min read
Mar 23, 2020Netflix Images Enhanced With AWS Lambdaby Mark Stier and Igor Okulist At Netflix we believe in empowering content creation and enabling new product experiences. Specifically for us it means eliminating the technical roadblocks for our studio and business partners to get artwork content to Netflix users. …Netflix5 min read
Published in Netflix TechBlog·Mar 13, 2020SVT-AV1: open-source AV1 encoder and decoderby Andrey Norkin, Joel Sole, Mariana Afonso, Kyle Swanson, Agata Opalach, Anush Moorthy, Anne Aaron — SVT-AV1 is an open-source AV1 codec implementation hosted on GitHub https://github.com/OpenVisualCloud/SVT-AV1/ under a BSD + patent license. As mentioned in our earlier blog post, Intel and Netflix have been collaborating on the SVT-AV1 encoder and decoder framework since August 2018. The teams have been working closely on SVT-AV1 development, discussing…Av 16 min read
Published in Netflix TechBlog·Mar 10, 2020Ready for changes with Hexagonal Architectureby Damir Svrtan and Sergii Makagon As the production of Netflix Originals grows each year, so does our need to build apps that enable efficiency throughout the entire creative process. Our wider Studio Engineering Organization has built numerous apps that help content progress from pitch (aka screenplay) to playback: ranging…Hexagonal Architecture9 min read
Published in Netflix TechBlog·Mar 3, 2020How Netflix uses Druid for Real-time Insights to Ensure a High-Quality ExperienceBy Ben Sykes Ensuring a consistently great Netflix experience while continuously pushing innovative technology updates is no easy feat. How can we be confident that updates are not harming our users? And that we’re actually making measurable improvements when we intend to? Using real-time logs from playback devices as a…Druid9 min read
Published in Netflix TechBlog·Feb 24, 2020Introducing DispatchBy Kevin Glisson, Marc Vilanova, Forest Monsen Netflix is pleased to announce the open-source release of our crisis management orchestration framework: Dispatch! Okay, but what is Dispatch? Put simply, Dispatch is: All of the ad-hoc things you’re doing to manage incidents today, done for you, and a bunch of other things you should’ve been doing, but have not had the time! Dispatch helps us effectively…Crisis Management6 min read
Published in Netflix TechBlog·Feb 14, 2020AVIF for Next-Generation Image CodingBy Aditya Mavlankar, Jan De Cock¹, Cyril Concolato, Kyle Swanson, Anush Moorthy and Anne Aaron TL; DR We need an alternative to JPEG that a) is widely supported, b) has better compression efficiency and c) has a wider feature set. We believe AV1 Image File Format (AVIF) has the potential. Using the…Encoding17 min read
Published in Netflix TechBlog·Feb 6, 2020Essential Suite — Artwork Producer AssistantBy: Hamid Shahid & Syed Haq Introduction Netflix continues to invest in content for a global audience with a diverse range of unique tastes and interests. Correspondingly, the member experience must also evolve to connect this global audience to the content that most appeals to each of them. Images that represent…Artwork8 min read
Published in Netflix TechBlog·Feb 5, 2020Netflix Now Streaming AV1 on AndroidBy Liwei Guo, Vivian Li, Julie Beckley, Venkatesh Selvaraj, and Jeff Watts Today we are excited to announce that Netflix has started streaming AV1 to our Android mobile app. AV1 is a high performance, royalty-free video codec that provides 20% improved compression efficiency over our VP9† encodes. …Android2 min read
Published in Netflix TechBlog·Jan 31, 2020Open-Sourcing riskquant, a library for quantifying riskNetflix has a program in our Information Security department for quantifying the risk of deliberate (attacker-driven) and accidental losses. This program started on the Detection Engineering team with a home-grown Python library called riskquant, which we’ve released as open source for you to use (and contribute to). Since that library…Security4 min read
Published in Netflix TechBlog·Dec 17, 2019DBLog: A Generic Change-Data-Capture FrameworkAndreas Andreakis, Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. …My SQL14 min read
Dec 12, 2019Rebooting datastores into the futureBy Josh Snyder The Netflix Cloud Data Engineering team is delighted to open source s3-flash-bootloader, our tool for performing in-place OS image upgrades on stateful cloud instances, substituting a new AMI where an old one was. …AWS6 min read
Published in Netflix TechBlog·Dec 10, 2019Netflix Hack Day — Fall 2019By Tom Richards, Carenina Garcia Motion, and Leslie Posada Hack Day at Netflix is an opportunity to build and show off a feature, tool, or quirky app. The goal is simple: experiment with new ideas/technologies, engage with colleagues across different disciplines, and have fun! We know even the silliest idea…Hackathons3 min read
Published in Netflix TechBlog·Dec 4, 2019Open-Sourcing Metaflow, a Human-Centric Framework for Data Scienceby David Berg, Ravi Kiran Chirravuri, Romain Cledat, Savin Goyal, Ferras Hamad, Ville Tuulos tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that…Data Science8 min read
Published in Netflix TechBlog·Dec 3, 2019Data Compression for Large-Scale Streaming ExperimentationJulie (Novak) Beckley, Andy Rhines, Jeffrey Wong, Matthew Wardrop, Toby Mao, Martin Tingley Ever wonder why Netflix works so well when you’re streaming at home, on the train, or in a foreign hotel? Behind the scenes, Netflix engineers are constantly striving to improve the quality of your streaming service. The…Data Science9 min read
Published in Netflix TechBlog·Nov 23, 2019Netflix at AWS re:Invent 2019by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Please stop by our “Living Room” for an opportunity to connect or reconnect with Netflixers. We’ve compiled our speaking events below so you know…AWS5 min read
Nov 15, 2019Garbage collecting unhealthy JVMs, a proactive approachBy Josh Snyder and Joseph Lynch Netflix’s Cloud Data Engineering team runs a variety of JVM applications, including popular datastores like Cassandra and Elasticsearch. …Java7 min read
Published in Netflix TechBlog·Nov 12, 2019Page Simulation for Better Offline Metrics at Netflixby David Gevorkyan, Mehmet Yilmaz, Ajinkya More, Gaurav Agrawal, Richard Wellington, Vivek Kaushal, Prasanna Padmanabhan, Justin Basilico At Netflix, we spend a lot of effort to make it easy for our members to find content they will love. To make this happen, we personalize many aspects of our service, including…Page Generation13 min read
Published in Netflix TechBlog·Nov 4, 2019GraphQL Search Indexingby Artem Shtatnov and Ravi Srinivas Ranganathan Almost a year ago we described our learnings from adopting GraphQL on the Netflix Marketing Tech team. We have a lot more to share since then! There are plenty of existing resources describing how to express a search query in GraphQL and paginate…Graph QL12 min read
Published in Netflix TechBlog·Oct 23, 2019Open-sourcing Polynote: an IDE-inspired polyglot notebookJeremy Smith, Jonathan Indig, Faisal Siddiqi We are pleased to announce the open-source launch of Polynote: a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more. Polynote provides data scientists and machine learning researchers with a notebook environment…Notebook11 min read
Published in Netflix TechBlog·Oct 21, 2019Open Sourcing Mantis: A Platform For Building Cost-Effective, Realtime, Operations-Focused ApplicationsBy Cody Rioux, Daniel Jacobson, Jeff Chao, Neeraj Joshi, Nick Mahilani, Piyush Goyal, Prashanth Ramdas, Zhenzhong Xu Today we’re excited to announce that we’re open sourcing Mantis, a platform that helps Netflix engineers better understand the behavior of their applications to ensure the highest quality experience for our members. …Observability5 min read
Published in Netflix TechBlog·Oct 18, 2019ML Platform Meetup: Infra for Contextual Bandits and Reinforcement LearningFaisal Siddiqi — Infrastructure for Contextual Bandits and Reinforcement Learning — theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches…Netflix9 min read
Published in Netflix TechBlog·Oct 16, 2019How Netflix microservices tackle dataset pub-subBy Ammar Khaku Introduction In a microservice architecture such as Netflix’s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as…Microservices13 min read
Published in Netflix TechBlog·Oct 15, 2019Delta: A Data Synchronization and Enrichment PlatformPart I: Overview — Andreas Andreakis, Falguni Jhaveri, Ioannis Papapanagiotou, Mark Cho, Poorna Reddy, Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), providing advanced search capabilities (ElasticSearch etc.), caching…Big Data9 min read
Published in Netflix TechBlog·Sep 23, 2019Evolving Regional EvacuationNiosha Behnam | Demand Engineering @ Netflix — At Netflix we prioritize innovation and velocity in pursuit of the best experience for our 150+ million global customers. …Microservices6 min read
Published in Netflix TechBlog·Sep 11, 2019Reimagining Experimentation Analysis at NetflixToby Mao, Sri Sri Perangur, Colin McFarland Another day, another custom script to analyze an A/B test. Maybe you’ve done this before and have an old script lying around. If it’s new, it’s probably going to take some time to set up, right? Not at Netflix. Suppose you’re running a…Experimentation8 min read
Sep 3, 2019Inca — Message Tracing and Loss Detection For Streaming Data @Netflixby Allen Wang, Real Time Data Infrastructure At Netflix, our real time data infrastructure have embraced the multi-cluster Kafka architecture and Flink powered stream processing platform, which enable us to deliver trillions of messages per day. This architecture is described in detail in the above blogs and is illustrated in…Stream Processing11 min read
Published in Netflix TechBlog·Aug 22, 2019Applying Netflix DevOps Patterns to WindowsBaking Windows with Packer By Justin Phelps and Manuel Correa Customizing Windows images at Netflix was a manual, error-prone, and time consuming process. In this blog post, we describe how we improved the methodology, which technologies we leveraged, and how this has improved service deployment and consistency. Artisan Crafted Images In the Netflix…Docker5 min read
Published in Netflix TechBlog·Jul 31, 2019Evolution of Netflix Conductor:v2.0 and beyond — By Anoop Panicker and Kishore Banala Conductor is a workflow orchestration engine developed and open-sourced by Netflix. If you’re new to Conductor, this earlier blogpost and the documentation should help you get started and acclimatized to Conductor. Netflix Conductor: A microservices orchestrator visibility into distributed workflowsmedium.com In the last two years since inception, Conductor has seen wide adoption and…Netflix9 min read
Jul 23, 2019Customer Call Centers: A UXer’s Secret WeaponGet in touch with the folks on the frontlines of product support for meaningful user insights — By Jen Schaefer As a Content Designer (aka UX Writer, Content Strategist) on Netflix’s Experience Design team, I spend a lot of time putting myself in the shoes of users to understand their pain points and moments of joy. How I do this on any given day ranges from informally…Design7 min read
Published in Netflix TechBlog·Jul 12, 2019Re-Architecting the Video GatekeeperBy Drew Koszewnik — This is the story about how the Content Setup Engineering team used Hollow, a Netflix OSS technology, to re-architect and simplify an essential component in our content pipeline — delivering a large amount of business value in the process. The Context Each movie and show on the Netflix service is carefully curated…Programming10 min read
Jul 9, 2019Scaling Appsec at NetflixBy Astha Singhal The Application Security team at Netflix is responsible for securing the applications that help run the Netflix business and the streaming product. Our customers are primarily engineering teams that produce software deployed within our cloud infrastructure. …Security4 min read
Published in Netflix TechBlog·Jul 2, 2019Bringing Rich Experiences to Memory-Constrained TV DevicesBy Jason Munning, Archana Kumar, Kris Range Netflix has over 148M paid members streaming on more than half a billion devices spanning over 1,900 different types. In the TV space alone, there are hundreds of device types that run the Netflix app. …Javascript Development13 min read
Jun 24, 2019Delightful User Interfaces: Easter EggsBy Cole Turner — The first easter egg appeared in the 1979 Atari release of Adventure the video game. Often hidden in the edges and corners of software, engineers have been leaving behind secret clues ever since. …CSS6 min read
Published in Netflix TechBlog·Jun 20, 2019Netflix Studio Hack Day — May 2019By Tom Richards, Carenina Garcia Motion, and Marlee Tart Hack Days are a big deal at Netflix. They’re a chance to bring together employees from all our different disciplines to explore new ideas and experiment with emerging technologies. For the most recent hack day, we channeled our creative energy towards…Hackathons3 min read
Published in Netflix TechBlog·Jun 4, 2019Predictive CPU isolation of containers at NetflixBy Benoit Rostykus, Gabriel Hartmann — Noisy Neighbors We’ve all had noisy neighbors at one point in our life. Whether it’s at a cafe or through a wall of an apartment, it is always disruptive. The need for good manners in shared spaces turns out to be important not just for people, but for your Docker containers too. …Machine Learning8 min read
Published in Netflix TechBlog·May 30, 2019Making our Android Studio Apps Reactive with UI Components & ReduxBy Juliano Moraes, David Henry, Corey Grunewald & Jim Isaacs Recently Netflix has started building mobile apps to bring technology and innovation to our Studio Physical Productions, the portion of the business responsible for producing our TV shows and movies. Our very first mobile app is called Prodicle and was…Reactive6 min read
Published in Netflix TechBlog·May 22, 2019Lerner — using RL agents for test case schedulingBy: Stanislav Kirdey, Kevin Cureton, Scott Rick, Sankar Ramanathan, Mrinal Shukla Introduction Netflix brings delightful customer experiences to homes on a variety of devices that continues to grow each day. Partners across the globe leverage Netflix device certification process on a continual basis to ensure that quality products and experiences are…Machine Learning8 min read
Published in Netflix TechBlog·May 1, 2019Rx onError GuidelinesBy Ed Ballot “Creating a good API is hard.” — anyone who has created an API used by others As with any API, wrapping your data stream in a Rx observable requires consideration for reasonable error handling and intuitive behavior. The following guidelines are intended to help developers create consistent…Rxjava4 min read
Published in Netflix TechBlog·May 1, 2019Engineering a Studio Quality Experience With High-Quality Audio at Netflixby Guillaume du Pontavice, Phill Williams and Kylee Peña (on behalf of our Streaming Algorithms, Audio Algorithms, and Creative Technologies teams) Remember the epic opening sequence of Stranger Things 2? The thrill of that car chase through Pittsburgh not only introduced a whole new set of mysteries, but it returned…9 min read
Published in Netflix TechBlog·Apr 29, 2019Python at NetflixBy Pythonistas at Netflix, coordinated by Amjith Ramanujam and edited by Ellen Livengood As many of us prepare to go to PyCon, we wanted to share a sampling of how Python is used at Netflix. We use Python through the full content lifecycle, from deciding which content to fund all…Data Science8 min read
Published in Netflix TechBlog·Apr 17, 2019Introducing SVT-AV1: a scalable open-source AV1 frameworkby Andrey Norkin, Joel Sole, Kyle Swanson, Mariana Afonso, Anush Moorthy, Anne Aaron — Netflix headquarters circa 2014. It’s a nice building with good architecture! This was the primary home of Netflix for a number of years during the company’s growth, but at some point Netflix had outgrown its home and needed more space. One approach to solve this problem would have been to…Av 17 min read
Apr 4, 2019Hailstorm — the Netflix scaling solution over Android TVby Rex Ching Hailstorm is an Android TV scaling program designed by Netflix to reduce the Netflix integration effort for partners on the Android TV platform. It aims to shorten partner’s time to market and to unlock more pay TV operators’ device reach opportunities, making Netflix even more accessible to…Android Tv Box6 min read
Published in Netflix TechBlog·Mar 25, 2019Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and EfficiencyBy: Di Lin, Girish Lingappa, Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question — “Can I run a check myself to understand what data is behind…Big Data8 min read
Published in Netflix TechBlog·Mar 21, 2019Netflix Public Bug Bounty, 1 year laterby Astha Singhal (Netflix Application Security) As Netflix continues to create entertainment people love, the security team continues to keep our members, partners, and employees secure. The security research community has partnered with us to improve the security of the Netflix service for the past few years through our responsible…Security3 min read
Published in Netflix TechBlog·Mar 12, 2019Spinnaker Sets Sail to the Continuous Delivery FoundationAuthor: Andy Glover Since releasing Spinnaker to the open source community in 2015, the platform has flourished with the addition of new cloud providers, triggers, pipeline stages, and much more. Myriad new features, improvements, and innovations have been added by an ever growing, actively engaged community. …Continuous Delivery3 min read
Published in Netflix TechBlog·Mar 8, 2019Design Principles for Mathematical Engineering in Experimentation Platform at NetflixJeffrey Wong, Senior Modeling Architect, Experimentation Platform Colin McFarland, Director, Experimentation Platform At Netflix, we have data scientists coming from many backgrounds such as neuroscience, statistics and biostatistics, economics, and physics; each of these backgrounds has a meaningful contribution to how experiments should be analyzed. To unlock these innovations we…Data Science7 min read
Published in Netflix TechBlog·Mar 6, 2019Trace Event, Chrome and More Profile Formats on FlameScopeLess than a year ago, FlameScope was released as a proof of concept for a new profile visualization. Since then, it helped us, and many other users, to easily find and fix performance issues, and allowed us to see patterns that we had never noticed before in our profiles. As…Performance4 min read
Published in Netflix TechBlog·Mar 6, 2019MezzFS — Mounting object storage in Netflix’s media processing platformBy Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. It’s used extensively in our media processing platform, which includes services like Archer and runs features like…Distributed Systems9 min read
Published in Netflix TechBlog·Mar 5, 2019How Data Inspires Building a Scalable, Resilient and Secure Cloud Infrastructure At NetflixBy: Jitender Aswani (Data Engineering and Infrastructure), Sebastien de Larquier (Science & Analytics) Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility and they are free to operate with freedom to satisfy their mission. This…Cloud Computing5 min read
Published in Netflix TechBlog·Feb 20, 2019Extending Vector with eBPF to inspect host and container performanceby Jason Koch, with Martin Spier, Brendan Gregg, Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. Today we are excited to announce latency…Java Script6 min read
Published in Netflix TechBlog·Feb 11, 2019Building a Cross-platform In-app Messaging Orchestration ServiceGeorge Abraham, Devika Chawla, Chris Beaumont, and Daniel Huang. — Thoughtful, relevant, and timely messaging is an integral part of a customer’s Netflix experience. The Netflix Messaging Engineering team builds the platform and the messages to communicate with Netflix customers. Messages in the Netflix App In-app messages at Netflix fall broadly into two channels — Notifications and Alerts. Notifications are content recommendations that show up…Messaging9 min read
Published in Netflix TechBlog·Feb 5, 2019Protecting a Story’s Future with History and ScienceBy Kylee Peña, Chris Clark, Mike Whipple, and Ben Sutor I — Kylee — have two photos from my parents’ wedding. Just two. This year they celebrated 40 years of marriage, so both photos were shot on film. Both capture a joy and awkwardness that come with young weddings. …Film And Television16 min read
Published in Netflix TechBlog·Feb 4, 2019Engineering to Improve Marketing Effectiveness (Part 3) — Scaling Paid Media campaignsThis is the third blog of the series on Marketing Technology at Netflix. This blog focuses on the marketing tech systems that are responsible for campaign setup and delivery of our paid media campaigns. The first blog focused on solving for creative development and localization at scale. …Advertising7 min read
Published in Netflix TechBlog·Feb 1, 2019Improving Experimentation Efficiency at Netflix with Meta Analysis and Optimal StoppingBy Gang Su & Ian Yohai From living rooms in Bogota, to morning commutes in Tokyo, to beaches in Los Angeles and dorms in Berlin, Netflix strives to bring joy to over 139 million members around the globe and connect people with stories they’ll love. Every bit of the customer…Data Science6 min read
Published in Netflix TechBlog·Jan 31, 2019Detecting Performance Anomalies in External Firmware Deploymentsby Richard Cool Netflix has over 139M members streaming on more than half a billion devices spanning over 1,700 different types of devices from hundreds of brands. This diverse device ecosystem results in a high dimensionality feature space, often with sparse data, and can make identifying device performance issues challenging…Data Science6 min read
Published in Netflix TechBlog·Dec 18, 2018Netflix OSS and Spring Boot — Coming Full CircleTaylor Wicksell, Tom Cellucci, Howard Yuan, Asi Bross, Noel Yap, and David Liu — In 2007, Netflix started on a long road towards fully operating in the cloud. Much of Netflix’s backend and mid-tier applications are built using Java, and as part of this effort Netflix engineering built several cloud infrastructure libraries and systems — Ribbon for load balancing, Eureka for service discovery, and…Java3 min read
Published in Netflix TechBlog·Dec 14, 2018Implementing the Netflix Media DatabaseIn the previous blog posts in this series, we introduced the Netflix Media DataBase (NMDB) and its salient “Media Document” data model. In this post we will provide details of the NMDB system architecture beginning with the system requirements — these will serve as the necessary motivation for the architectural…Nmdb19 min read
Published in Netflix TechBlog·Dec 13, 2018Performance comparison of video coding standards: an adaptive streaming perspectiveby Joel Sole, Liwei Guo, Andrey Norkin, Mariana Afonso, Kyle Swanson, Anne Aaron — “This is my advice to people: Learn how to cook, try new recipes, learn from your mistakes, be fearless, and above all have fun” — Julia Child (American chef, author, and television personality) At Netflix, we are continually refining the recipes we use to serve your favorite shows and movies…Video10 min read
Published in Netflix TechBlog·Dec 12, 2018Modernizing the Web Playback UIby Corey Grunewald & Matt Jaquish — Since 2013, the user experience of playing videos on the Netflix website has changed very little. During this period, teams at Netflix have rolled out amazing video playback features, but the visual design and user controls of the playback UI have remained the same. Over the past two years, the…Java Script11 min read
Published in Netflix TechBlog·Dec 10, 2018Our learnings from adopting GraphQLA Marketing Tech Campaign — by Artem Shtatnov and Ravi Srinivas Ranganathan In an earlier blog post, we provided a high-level overview of some of the applications in the Marketing Technology team that we build to enable scale and intelligence in driving our global advertising, which reaches users on sites like The New York Times…Graph QL9 min read
Published in Netflix TechBlog·Dec 4, 2018Cache warming: Agility for a stateful serviceby Deva Jayaraman, Shashi Madappa, Sridhar Enugula, and Ioannis Papapanagiotou EVCache has been a fundamental part of the Netflix platform (we call it Tier-1), holding Petabytes of data. Our caching layer serves multiple use cases from signup, personalization, searching, playback, and more. It is comprised of thousands of nodes in…AWS9 min read
Published in Netflix TechBlog·Nov 29, 2018Netflix Information Security: Preventing Credential Compromise in AWSby Will Bengtson Previously we wrote about a method for detecting credential compromise in your AWS environment. The methodology focused on a continuous learning model and first use principle. This solution still is reactive in nature — we only detect credential compromise after it has already happened.. …AWS9 min read
Published in Netflix TechBlog·Nov 27, 2018Netflix at AWS re:Invent 2018by Shaun Blackburn AWS re:Invent is back in Las Vegas this week! Many Netflix engineers and leaders will be among the 40,000 attending the conference to connect with fellow cloud and OSS enthusiasts. You can find us at our booth on the expo floor, speaking on a variety of subjects…Netflixsecurity6 min read
Published in Netflix TechBlog·Nov 12, 2018Delivering Meaning with Previews on WebBy Corey Grunewald and Tony Casparro — As the Netflix catalog of films and series continues to grow, it becomes more challenging to present members with enough information to decide what to watch. How can a member tell if a movie is both a horror and a comedy? …Web Development6 min read
Published in Netflix TechBlog·Nov 7, 2018Netflix Hack Day — Fall 2018by Tom Richards, Carenina Motion, Ruslan Meshenberg, Leslie Posada, and Kaely Coon Hack Days are a big deal at Netflix. They’re a chance to bring together employees from all our different disciplines to explore new ideas and experiment with emerging technologies. This Hack Day, there were hacks that ranged…Netflix2 min read
Published in Netflix TechBlog·Nov 5, 2018Scaling Time Series Data Storage — Part IIby Dhruv Garg, Dhaval Patel, Ketan Duvedi — In January 2016 Netflix expanded worldwide, opening service to 130 additional countries and supporting 20 total languages. Later in 2016 the TV experience evolved to include video previews during the browsing experience. More members, more languages, and more video playbacks stretched the times series data storage architecture from part 1…Big Data9 min read
Published in Netflix TechBlog·Oct 31, 2018Netflix Media Database — the Media Timeline Data ModelIn the previous post in this series, we described some important Netflix business needs as well as traits of the media data system — called “Netflix Media DataBase” (NMDB) that is used to address them. The curious reader might have noticed that a majority of these characteristics relate to properties…Nmdb10 min read
Published in Netflix TechBlog·Oct 30, 2018The New Netflix Stethoscope Native AppA user focused approach to endpoint health and management — We are happy to announce the next big release in user focused security, the Stethoscope native app. The new native app includes basic device health recommendations with inline clickable instructions on how to update settings. …Endpoint Security5 min read
Published in Netflix TechBlog·Oct 26, 2018VMAF: The Journey Continuesby Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy and Jan De Cock How will Netflix members rate the quality of this video — poor, average or excellent? Which video clip looks better — encoded with Codec A or Codec B? For this episode, at 1000 kbps…Machine Learning15 min read
Published in Netflix TechBlog·Oct 17, 2018Lumen: Custom, Self-Service Dashboarding For NetflixBy Trent Willis — Netflix generates a lot of data. One of the ways that we gain useful insights is by visualizing that data in dashboards which allow us to comprehend large amounts of information quickly. …Data Visualization8 min read
Published in Netflix TechBlog·Oct 15, 2018The Netflix Media DatabaseLet us imagine we are working on the next generation adaptive video streaming algorithm. Our goal is to minimize the playback startup time for the millions of Netflix members all over the world. To do that we need to gather aggregate statistics (minimum, maximum, median, mean, any number of percentiles)…Distributed Systems7 min read
Published in Netflix TechBlog·Sep 29, 2018Rethinking Netflix’s Edge Load BalancingBy Mike Smith, Netflix Cloud Gateway We briefly touched on some of the load balancing improvements we’ve recently been making in our Open Sourcing Zuul 2 post. In this post, we’ll go into more detail on the whys, hows and results of that work. Goals On the Netflix Cloud Gateway team…Load Balancing16 min read
Published in Netflix TechBlog·Sep 26, 2018Netflix’s Production Technology = Voltronby Chris Goss Change management is hard. In everyday production, there are numerous factors working against embracing change. Limited preparation time, whole new show = whole new crew, innumerable planning variables, and the challenge of driving an operational plan based on creative instincts. These are problems that technology is not…Saa S8 min read
Published in Netflix TechBlog·Sep 25, 2018Enhancing the Netflix UI Experience with HDRby Yuji Mano, Benbuck Nason, Joe Drago Some of you have probably heard about HDR, or High Dynamic Range, which can produce a higher level of brightness, a wider range of colors, and an increased level of detail and accuracy that empowers visual storytellers to deliver more impactful experiences for…Hdr17 min read
Published in Netflix TechBlog·Sep 19, 2018Quasi Experimentation at NetflixColin McFarland, Michael Pow, Julia Glick Experimentation informs much of our decision making at Netflix. We design, analyze, and execute experiments with rigor so that we have confidence that the changes we’re making are the right ones for our members and our business. We have many years of experience running…Experimentation5 min read
Published in Netflix TechBlog·Sep 18, 2018Netflix at the Spinnaker Summit 2018Author: Andy Glover Since releasing Spinnaker as an open source project in late 2015, the Spinnaker user and contributor community has grown by leaps and bounds. Our strong partnership with Google and other companies has led to a plethora of innovations and features that enable rapid, safe, and reliable delivery…Cloud Computing2 min read
Published in Netflix TechBlog·Sep 17, 2018Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significanceby Martin Tingley Streaming video experimentation at Netflix seeks to optimize the Quality of Experience (QoE for short) of the videos we stream to our 130 million members around the world. To measure QoE, we look at a wide variety of metrics for each playback session, including play delay; the…Data Science9 min read
Published in Netflix TechBlog·Sep 14, 2018Announcing the Netflix Post Technology Allianceby Chris Fetner and Richard Smith You’re working on the next great show or film that will deliver to Netflix. You know Netflix has a high quality bar for its content and you want the delivery to go smoothly. Do you know if the tools you have invested in can…Technology3 min read
Published in Netflix TechBlog·Sep 10, 2018Keystone Real-time Stream Processing PlatformKeystone Stream Processing Platform is Netflix’s data backbone and an essential piece of infrastructure that enables engineering data-driven culture. While Keystone focuses on data analytics, it is worth mentioning there is another Netflix homegrown reactive stream processing platform called Mantis that targets operational use cases. We’ll discuss Mantis and its…Big Data13 min read
Published in Netflix TechBlog·Sep 4, 2018Engineering to Improve Marketing Effectiveness (Part 2) — Scaling Ad Creation and Managementby Ravi Srinivas Ranganathan, Gopal Krishnan In the first part of this series of blogs, we described our philosophy, motivations, and approach to blending ad technology into our marketing. In addition, we laid out some of the engineering undertakings to solve creative development and localization at scale. In this Part-2…Marketing10 min read
Published in Netflix TechBlog·Aug 23, 2018Part 2: Scheduling Notebooks at Netflixby Matthew Seal, Kyle Kelley, and Michelle Ufford At Netflix we’ve put substantial effort into adopting notebooks as an integrated development platform. The idea started as a discussion of what development and collaboration interfaces might look like in the future. It evolved into a strategic bet on notebooks, both as…Jupyter11 min read
Published in Netflix TechBlog·Aug 16, 2018Beyond Interactive: Notebook Innovation at NetflixBy Michelle Ufford, M Pacer, Matthew Seal, and Kyle Kelley Notebooks have rapidly grown in popularity among data scientists to become the de facto standard for quick prototyping and exploratory analysis. At Netflix, we’re pushing the boundaries even further, reimagining what a notebook can be, who can use it, and…Data Science13 min read
Published in Netflix TechBlog·Aug 8, 2018Netflix Cloud Security: Detecting Credential Compromise in AWSWill Bengtson, Netflix Security Tools and Operations Credential compromise is an important concern for anyone operating in the cloud. The problem becomes more obvious over time, as organizations continue to adopt cloud resources as part of their infrastructure without maintaining an accompanying capability to observe and react to these compromises…AWS7 min read
Published in Netflix TechBlog·Aug 6, 2018Pseudo Localization @ Netflixby Tim Brandall In the past 8 years Netflix has transformed from an English-only product, to now supporting 26 languages and growing. As we add language support for our members residing in 190 different countries, scaling globalization at Netflix has never been more important. Along the way we’ve built out…Localization6 min read
Published in Netflix TechBlog·Jul 18, 2018Netflix Cloud Security SIRT releases Diffy: A Differencing Engine for Digital Forensics in the CloudForest Monsen and Kevin Glisson, Netflix Security Intelligence and Response Team The Netflix Security Intelligence and Response Team (SIRT) announces the release of Diffy under an Apache 2.0 license. …Dev Ops5 min read
Published in Netflix TechBlog·Jul 13, 2018Evolution of Application Data Caching : From RAM to SSDIn 2013, we introduced EVCache a distributed in-memory caching solution based on memcached that offers low-latency, high-reliability caching and storage. It is well integrated with AWS and EC2, a Netflix OSS project, and in many occasions it is termed as “the hidden microservice”. …Cloud Computing6 min read
Published in Netflix TechBlog·Jul 10, 2018Auto Scaling Production Services on TitusBy Andrew Leung, Amit Joshi, and the rest of the Titus team. Over the past three years, Netflix has been investing in container technology. A large part of this investment has been around Titus, Netflix’s container management platform that was open sourced in April of 2018. …AWS5 min read
Published in Netflix TechBlog·Jun 21, 2018Netflix at Spark+AI Summit 2018By Faisal Siddiqi A glimpse at Spark usage for Netflix Recommendations Apache Spark has been an immensely popular big data platform for distributed computing. Netflix has been using Spark extensively for various batch and stream-processed workloads. A substantial list of use cases for Spark computation come from the various applications in the domain of content recommendations and personalization…Big Data5 min read
Published in Netflix TechBlog·Jun 20, 2018Simplifying Media Innovation at Netflix with ArcherBy Naveen Mareddy, Frank San Miguel, Rick Wong, Mangala Prabhu and Olof Johansson Imagine you are developing a quick prototype to sift through all the frames of the movie Bright to find the best shots of Will Smith with an action-packed background. …Docker8 min read
Published in Netflix TechBlog·Jun 18, 2018Growth Engineering at Netflix — Accelerating Innovationby Eric Eiswerth Millions of people visit Netflix every day. Many of them are already Netflix members, looking to enjoy their favorite movies and TV shows, and we work hard to ensure they have a great experience. …UX6 min read
Published in Netflix TechBlog·Jun 14, 2018Metacat: Making Big Data Discoverable and Meaningful at Netflixby Ajoy Majumdar, Zhen Li Most large companies have numerous data sources with different data formats and large data volumes. These data stores are accessed and analyzed by many people throughout the enterprise. At Netflix, our data warehouse consists of a large number of data sets stored in Amazon S3…Big Data7 min read
Published in Netflix TechBlog·Jun 13, 2018The End of Video Coding?By Anne Aaron and Jan De Cock — In the IEEE Signal Processing Magazine issue November 2006 article “Future of Video Coding and Transmission” Prof. Edward Delp started by asking the panelists “Is video coding dead? Some feel that, with the higher coding efficiency of the H.264/MPEG-4 . . . perhaps there is not much more to do…Video Encoding6 min read
Published in Netflix TechBlog·Jun 4, 2018Lessons from Building Observability Tools at NetflixOur mission at Netflix is to deliver joy to our members by providing high-quality content, presented with a delightful experience. We are constantly innovating on our product at a rapid pace in pursuit of this mission. Our innovations span personalized title recommendations, infrastructure, and application features like downloading and customer…Observability9 min read
Published in Netflix TechBlog·May 21, 2018Open Sourcing Zuul 2We are excited to announce the open sourcing of Zuul 2, Netflix’s cloud gateway. We use Zuul 2 at Netflix as the front door for all requests coming into Netflix’s cloud infrastructure. Zuul 2 significantly improves the architecture and features that allow our gateway to handle, route, and protect Netflix’s…Zuul7 min read
Published in Netflix TechBlog·May 17, 2018Full Cycle Developers at Netflix — Operate What You BuildThe year was 2012 and operating a critical service at Netflix was laborious. Deployments felt like walking through wet sand. Canarying was devolving into verifying endurance (“nothing broke after one week of canarying, let’s push it”) rather than correct functionality. Researching issues felt like bouncing a rubber ball between teams…Dev Ops10 min read
Published in Netflix TechBlog·May 15, 2018Multi-Cloud Continuous Delivery with Spinnaker report now available.by Emily Burns, Asher Feldman, Rob Fletcher, Tomas Lin, Justin Reynolds, Chris Sanden and Rob Zienert We’re pleased to announce the release of our O’Reilly report, Continuous Delivery with Spinnaker. The report is available to download for free on the Spinnaker website. ( Pdf | Epub | Mobi ) About The Report At…Continuous Delivery2 min read
Published in Netflix TechBlog·May 14, 2018Netflixにおける日本語字幕の導入(Please note that this article is a localized (to Japanese) version of a corresponding tech blog article in the English language) Netflixでは、2015年9月の日本における配信サービス開始時から日本語字幕を提供して …日本語字幕36 min read
Published in Netflix TechBlog·May 11, 2018Engineers Making Movies (AKA Open Source Test Content)by Andy Schuler and Matthew Donato If you’re a die-hard Netflix fan, you may have already stumbled onto the dark and brooding story of a timeless struggle between man and nature, water and fountain, ball and tether known much more plainly as Example Short. In the good old days of…Technology7 min read
Published in Netflix TechBlog·May 9, 2018Announcing the Netflix Research WebsiteWe’re pleased to announce that we’ve launched a new website at research.netflix.com that provides an overview of the research that we do here at Netflix. We have many amazing researchers working on a variety of hard problems and are happy to share some of our work with the world. Netflix…Research3 min read
Apr 27, 2018Tips for High AvailabilityBy Andy Glover and Katharina Probst Over the past four years, Netflix has gone from less than 50 Million subscribers to 125 Million subscribers. While this kind of growth has caused us no shortage of scaling challenges, we actually managed to improve the overall availability of our service in that…Continuous Delivery9 min read
Published in Netflix TechBlog·Apr 18, 2018Titus, the Netflix container management platform, is now open sourceby Amit Joshi, Andrew Leung, Corin Dwyer, Fabio Kung, Sargun Dhillon, Tomasz Bak, Andrew Spyker, Tim Bozarth Today, we are open-sourcing Titus, our container management platform. Titus powers critical aspects of the Netflix business, from video streaming, recommendations and machine learning, big data, content encoding, studio technology, internal engineering tools…Netflixoss7 min read
Published in Netflix TechBlog·Apr 10, 2018Automated Canary Analysis at Netflix with Kayentaby Michael Graff and Chris Sanden Today, in partnership with Google, we have open sourced Kayenta, a platform for Automated Canary Analysis (ACA). Kayenta leverages lessons learned over the years of delivering rapid and reliable changes into production at Netflix. It is a crucial component of delivery at Netflix as…Automated Testing8 min read
Published in Netflix TechBlog·Apr 4, 2018Netflix FlameScopeWe’re excited to release FlameScope: a new performance visualization tool for analyzing variance, perturbations, single-threaded execution, application startup, and other time-based issues. It has been created by the Netflix cloud performance engineering team and just released as open source, and we welcome help from others to develop the project further…Performance6 min read
Published in Netflix TechBlog·Apr 2, 2018Women Living on the Edge Eventby Karen Casella, Sangeeta Narayanan, Kim Trott, Gaya Varadarajan, Haripriya Murthy Netflix’s Edge Engineering team recently hosted a Women in Technology event where we shared some of the challenges that come with delivering over 1 billion streaming hours per week to delight more than 100M users around the globe. We welcomed senior women technologists and allies to our campus for networking and dinner at Netflix HQ.Tech1 min read
Published in Netflix TechBlog·Mar 27, 2018Data Science and the Art of Producing Entertainment at Netflixby Ritwik Kumar, Vinith Misra, Jen Walraven, Lavanya Sharan, Bahareh Azarnoush, Boris Chen, Nirmal Govind Netflix has released hundreds of Originals and plans to spend $8 billion over the next year on content. Creators of these stories pour their hearts and souls into turning ideas into joy for our viewers…Film10 min read
Mar 24, 2018Performance Under LoadAdaptive Concurrency Limits @ Netflix by Eran Landau, William Thurston, Tim Bozarth At Netflix we are obsessed with service availability, and we’ve written several blog posts over the years about how we achieve our goals. These techniques include circuit breakers, concurrency limits, chaos testing and more. Today we’re announcing our…Programming5 min read
Published in Netflix TechBlog·Mar 23, 2018Using Machine Learning to Improve Streaming Quality at Netflixby Chaitanya Ekanadham One of the common questions we get asked is: “Why do we need machine learning to improve streaming quality?” This is a really important question, especially given the recent hype around machine learning and AI which can lead to instances where we have a “solution in search…Machine Learning7 min read
Published in Netflix TechBlog·Mar 21, 2018Launching the Netflix Public Bug Bounty Programby Sunil Agrawal, Scott Behrens, Dave King, Astha Singhal, Patrick Thomas, Andy Hoernecke, Madan Sriraman — Netflix’s goal is to deliver joy to our 117+ million members around the world, and it’s the security team’s job to keep our members, partners and employees secure. We have been engaging with the security community to achieve this goal through programs like responsible disclosure and private bug bounty over…Security3 min read
Mar 16, 2018Scaling Image Validation Across Multiple Platformsby Natasha Lee — Introduction Our Team — the Test, Tools, and Reliability (TTR) team — is responsible for validating the quality of the Netflix SDK which we make available to our partners (Samsung, LG, Sony, Vizio, Roku, PlayStation, Xbox, etc…). For context, the core make up of our SDK includes the scripting engine, the…Automation Tools9 min read
Published in Netflix TechBlog·Mar 14, 2018Engineering to Improve Marketing Effectiveness (Part 1)Authored by — Subir Parulekar, Gopal Krishnan — “Make people so excited about our content that they sign up and watch” - Kelly Bennett, Netflix Chief Marketing Officer This statement has become the driving force on our Advertising Technology (AdTech) team. With a slate of great original content to promote, Netflix has a unique opportunity to use both…Marketing9 min read
Published in Netflix TechBlog·Mar 13, 2018Netflix Hack Day — Winter 2018by Tom Richards, Ruslan Meshenberg, Leslie Posada, Daniel Jacobson, Kaely Coon About a week ago, Netflix hosted another successful Hack Day. This event gives Netflix employees a chance to take a break from everyday work, collaborate with new people, and experiment with new technologies. …Hackday3 min read
Published in Netflix TechBlog·Mar 12, 2018Project Nimble: Region Evacuation ReimaginedWe are proud to present Nimble: the evolution of the Netflix failover architecture that makes region evacuation an order of magnitude faster. At Netflix, our goal is to be there for our customers whenever they want to come and watch their favorite shows. A lot of the work we do…AWS9 min read
Published in Netflix TechBlog·Mar 9, 2018Optimized shot-based encodes: Now Streaming!By Megha Manohara, Anush Moorthy, Jan De Cock, Ioannis Katsavounidis and Anne Aaron — Bad picture quality — blockiness, blurring, distorted faces and objects — can draw you out of that favorite TV show or movie you’re watching on Netflix. In many cases, low bandwidth networks or data caps prevent us from delivering the perfect picture. To address this, the Netflix Video Algorithms team…Video Encoding8 min read
Published in Netflix TechBlog·Mar 5, 2018Dynamic optimizer — a perceptual video encoding optimization frameworkBy Ioannis Katsavounidis, Sr. Research Scientist, Video Algorithms — Motivation Video encoding has fueled academic research over the past 25 years and enabled compelling products and services. Many companies are built around video encoding and transmission — Netflix and Google’s YouTube are two prime examples of video-centric companies. The fundamentals of video encoding haven’t changed for all these years, as…Video Encoding16 min read
Published in Netflix TechBlog·Feb 7, 2018AVA: The Art and Science of Image Discovery at NetflixAuthored by — Madeline, Lauren, Boris, Tim, Parth, Eugene and Apurva Introduction At Netflix, the Content Platform Engineering and Global Product Creative teams know that imagery plays an incredibly important role in how viewers find new shows and movies to watch. We take pride in surfacing the unique elements of a…Machine Learning7 min read
Published in Netflix TechBlog·Feb 1, 2018Fast JSON API serialization with Ruby on Railsby Shishir Kakaraddi, Srinivas Raghunathan, Adam Gross and Ryan Johnston We are pleased to announce the open source release of the Fast JSON API gem geared towards Ruby on Rails applications. Introduction Fast JSONAPI is aimed at providing all the major functionality that Active Model Serializer (AMS) provides, along with…Ruby On Rails2 min read
Published in Netflix TechBlog·Jan 24, 2018Scaling Time Series Data Storage — Part Iby Ketan Duvedi, Jinhua Li, Dhruv Garg, Philip Fisher-Ogden — Introduction The growth of internet connected devices has led to a vast amount of easily accessible time series data. Increasingly, companies are interested in mining this data to derive useful insights and make data-informed decisions. Recent technology advancements have improved the efficiency of collecting, storing and analyzing time series data, spurring…Compression9 min read
Published in Netflix TechBlog·Dec 20, 2017Netflix Now Supports HDR on Windows 10by Maria Kazandjieva, Greg Freedman, Matt Trunnell — We are thrilled to announce the addition of High Dynamic Range (HDR) support on Windows 10 for both the Edge browser and the Netflix app. With this update, Netflix members who have a supported device and a premium plan can enjoy amazing Netflix movies and shows in HDR. With HDR…Windows 102 min read
Published in Netflix TechBlog·Dec 20, 2017Integrating imperative APIs into a React applicationby Sriram Rao The Netflix TV app plays videos not only when users choose to watch a title, but also while browsing to find something great to enjoy. …Java Script6 min read
Published in Netflix TechBlog·Dec 12, 2017Distributing Content to Open ConnectBy Mohit Vora, Andrew Berglund, Videsh Sadafal, David Pfitzner, and Ellen Livengood — In previous posts we’ve talked about how we calculate, predict, and use content popularity for Open Connect to maximize the efficiency of our content delivery network and other data science challenges in this space. We also recently talked about improvements we made in the server throughput space. In this blog…Open Connect6 min read
Published in Netflix TechBlog·Dec 11, 2017Implementing Japanese Subtitles on NetflixJapanese subtitles were first made available on the Netflix service as a part of the Japanese launch in September 2015. This blog post provides a technical description of the work we did leading up to this launch. We cover topics including our specification for source subtitle files, the transformation model…Subtitles15 min read
Published in Netflix TechBlog·Dec 7, 2017Artwork Personalization at NetflixBy Ashok Chandrashekar, Fernando Amat, Justin Basilico and Tony Jebara For many years, the main goal of the Netflix personalized recommendation system has been to get the right titles in front each of our members at the right time. With a catalog spanning thousands of titles and a diverse member…Machine Learning13 min read
Published in Netflix TechBlog·Nov 30, 2017Innovating Faster on Personalization Algorithms at Netflix Using InterleavingBy Joshua Parks, Juliette Aurisset, Michael Ramm The Netflix experience is powered by a family of ranking algorithms, each optimized for a different purpose. For instance, the Top Picks row on the homepage makes recommendations based on a personalized ranking of videos, and the Trending Now row also incorporates recent…Data Science8 min read
Published in Netflix TechBlog·Nov 20, 2017Making it easier to contribute to Netflix OSSby Travis McPeak and Andrew Spyker Contributing to open source software can be a very rewarding experience that creates opportunities to learn about new problems and technologies, apply problem solving skills, meet and work with new people, and join a community pursuing a common goal. …Open Source2 min read
Published in Netflix TechBlog·Nov 14, 2017Updates on Netflix’s Container Management PlatformWe continue to share lessons learned from scheduling and executing containers in production at scale. This blog posts summarizes not only recent publications on our container management platform (Project Titus) but also future collaboration opportunities. Publications We were honored when the Association for Computing Machinery (ACM) asked for us to…Docker3 min read
Published in Netflix TechBlog·Nov 6, 2017Netflix at AWS re:Invent 2017by Jason Chan Netflix is excited to be heading back to Las Vegas for AWS re:Invent at the end of the month! Many Netflix engineers and recruiters will be in attendance, and we’re looking forward to meeting and reconnecting with cloud enthusiasts and Netflix OSS users. We’re posting the schedule…Cloud Computing11 min read
Published in Netflix TechBlog·Oct 18, 2017Machine Learning Platform meetupRecap of the Oct 2017 ML Platform meetup at Netflix HQ — By Faisal Siddiqi Machine Learning is making fast inroads into many areas of business and is being employed in an increasingly widening array of commercial applications. At Netflix, ML has been used for several years on our Recommendations and Personalization problems. Over the last couple of years, ML has expanded…Machine Learning7 min read
Published in Netflix TechBlog·Oct 13, 2017Developer Experience Lessons Operating a Serverless-like Platform At Netflix — Part IIBy Ludovic Galibert, Vasanth Asokan and Sangeeta Narayanan In Part 1 of this series, we outlined key learnings the Edge Developer Experience team gained from operating the API dynamic scripting platform which provides a serverless or FaaS like experience for client application developers. We addressed the concerns around getting code…Continuous Delivery7 min read
Published in Netflix TechBlog·Sep 29, 2017Serving 100 Gbps from an Open Connect ApplianceBy Drew Gallatin — In the summer of 2015, the Netflix Open Connect CDN team decided to take on an ambitious project. …Content Delivery Network15 min read
Published in Netflix TechBlog·Sep 18, 2017Scaling Event Sourcing for Netflix Downloads, Episode 2by Karen Casella, Phillipa Avery, Robert Reta, Joseph Breuer In the first episode of this series of posts, we introduced the Netflix downloads project and the use cases that led us to consider a solution based on the event sourcing pattern. In this post, we provide an overview of the…Software Architecture9 min read
Published in Netflix TechBlog·Sep 11, 2017Scaling Event Sourcing for Netflix Downloads, Episode 1by Karen Casella, Phillipa Avery, Robert Reta, Joseph Breuer Early in 2016, several Netflix teams were asked the question: “What would it take to allow members to download and view content offline on their mobile devices?” For the Playback Licensing team, this meant that we needed to provide a content…Netflix6 min read
Published in Netflix TechBlog·Aug 29, 2017Netflix Hack Day — Summer 2017by Daniel Jacobson, Ruslan Meshenberg, Leslie Posada, and Tom Richards About a week ago, Netflix hosted another great Hack Day. The event gives Netflix employees a chance to take a break from everyday work, have fun, experiment with new technologies, and collaborate with new people. Like previous Hack Days, we…Raspberry Pi3 min read
Published in Netflix TechBlog·Aug 21, 2017A Brief History of Open Source from the Netflix Cloud Security Teamby Jason Chan This summer marks three years of releasing open source software for the Netflix Cloud Security team. It’s been a busy three years — our most recent release marks 15 open source projects — so we figured a roundup and recap would be useful. Penetration testing tools, vulnerabilities…AWS5 min read
Aug 2, 2017Introducing Vectorflowa lightweight neural network library for sparse data — By Benoît Rostykus Introduction With the deluge of deep learning libraries and software innovation in the field over the last few years, it is an exciting time to be working on machine learning problems. …Machine Learning5 min read
Published in Netflix TechBlog·Jul 28, 2017Starting the AvalancheApplication DDoS In Microservice Architectures — By Scott Behrens and Bryan Payne Introduction We’d like to introduce you to one of the most devastating ways to cause service instability in modern micro-service architectures: application DDoS. A specially crafted application DDoS attack can cause cascading system failures often for a fraction of the resources needed to conduct a…Microservices9 min read
Published in Netflix TechBlog·Jul 26, 2017ChAP: Chaos Automation PlatformWe are excited to announce ChAP, the newest member of our chaos tooling family! Chaos Monkey and Chaos Kong ensure our resilience to instance and regional failures, but threats to availability can also come from disruptions at the microservice level. …Chaos Engineering5 min read
Jul 20, 2017Persisting Falcor on AndroidEveryone hates waiting, right? This can be especially maddening when using an application on a slow network. Not all of the 100M+ Netflix users all over the world live in areas with access to reliable, high-speed internet. The Android application team here at Netflix has been working on several projects…Android5 min read
Published in Netflix TechBlog·Jul 11, 2017Developer Experience Lessons Operating a Serverless-like Platform At NetflixBy Vasanth Asokan, Ludovic Galibert and Sangeeta Narayanan The Netflix API is based on a dynamic scripting platform that handles thousands of changes per day. This platform allows our client developers to create a customized API experience on over a thousand device types by executing server side adapter code in…Continuous Delivery7 min read
Published in Netflix TechBlog·Jul 5, 2017Netflix Platform Engineering — we’re just getting startedby Ruslan Meshenberg — “Aren’t you done with every interesting challenge already?” I get this question in various forms a lot. During interviews. At conferences, after we present on some of our technologies and practices. At meetups and social events. You have fully migrated to the Cloud, you must be done… You created multi-regional…Microservices4 min read
Published in Netflix TechBlog·Jun 20, 2017Content Popularity for Open ConnectBy Mohit Vora, Lara Deek, Ellen Livengood — There are many reasons why Netflix cares about the popularity of our TV shows and movies. On the Open Connect Content Delivery team, we predict content popularity to maximize our infrastructure efficiency. Some months ago, we blogged about how we use proactive caching to keep the content on our global…Content Delivery Network7 min read
Published in Netflix TechBlog·Jun 13, 2017A/B Testing and Beyond: Improving the Netflix Streaming Experience with Experimentation and Data Scienceby Nirmal Govind Golden Globes for The Crown. An Oscar win for The White Helmets. It’s an exciting time to be a Netflix member, with the power to stream such incredible content on a Smart TV or a mobile phone, from the comfort of one’s home or on a daily…Data Science13 min read
Published in Netflix TechBlog·Jun 7, 2017Evolving the Netflix Data Platform with Genie 3by Tom Gianos The big data space continues to change at a rapid pace. Data scientists and analysts have more tools than ever at their disposal whether it be Spark, R, Presto, or traditional engines like Hive and Pig. At Netflix the Big Data Platform team is responsible for making…Big Data7 min read
Published in Netflix TechBlog·Jun 6, 2017Simone - A Distributed Simulation ServiceBy Satyajit Thadeshwar, Mayank Agarwal, Sangeeta Narayanan & Kevin Lew Hundreds of models of smart TVs, game consoles, mobile devices, and other video streaming devices get shipped with a Netflix app pre-installed. Before shipping these devices, manufacturers need to have the app certified on their device firmware. …Microservices8 min read
Published in Netflix TechBlog·Jun 5, 2017Introducing Aardvark and RepokidAWS Least Privilege for Distributed, High-Velocity Development — by Jason Chan, Patrick Kelley, and Travis McPeakAWS5 min read
Published in Netflix TechBlog·May 27, 2017Spinnaker OrchestrationAuthor: Rob Fletcher When the Spinnaker project first started more than two years ago we implemented Orca — Spinnaker’s orchestration engine µservice — using Spring Batch. It wasn’t an entirely unreasonable fit at the time. It gave us atomic, compartmentalized units of work (tasks in a Spinnaker pipeline), retry semantics…Spinnaker7 min read
Published in Netflix TechBlog·May 26, 2017How Data Science Helps Power Worldwide Delivery of Netflix Contentby Andrew Berglund Have you ever wondered where your video comes from when you watch Netflix? We serve video streams out of our own content delivery network (CDN), called Open Connect, which is tailored to one specific application: delivering internet TV to our members around the world. This system is…Data Science6 min read
Published in Netflix TechBlog·May 23, 2017Automation as a Service — Introducing ScriptflaskAuthors : Fazal Allanabanda and Vilas Veeraraghavan Introduction Less than a year ago, we shared our testing wins and the challenges that lie ahead while testing High Impact Titles (HITs) globally. All this while ensuring the pace of innovation at Netflix does not slow down. Since then, we have made significant…Integration Testing11 min read
Published in Netflix TechBlog·May 18, 2017Pass the Remote: User Input on TV Devicesby Andrew Eichacker The Netflix TV team works with device manufacturers to explore new input methods (like your phone!) and improve the screens we watch our favorite shows on. Beyond that, we’re testing the boundaries for content discovery and playback while bringing Netflix to more users around the world. We’ve…Java Script7 min read
Published in Netflix TechBlog·May 5, 2017Netflix OSS: Batch Requests with Ruby on Rails and Ember.jsby Srinivas Raghunathan, Shishir Kakaraddi, and James Menera We are pleased to announce the open source release of three Batch Request related libraries. Libraries: Ember Batch Request (Ember JS Add on) Batch Request API (Rails Middleware) Batch Request Client (Ruby Client) Introduction Batching allows you to pass several operations in…Java Script2 min read
Published in Netflix TechBlog·Apr 27, 2017Towards true continuous integration: distributed repositories and dependenciesFor the past 8 years, Netflix has been building and evolving a robust microservice architecture in AWS. Throughout this evolution, we learned how to build reliable, performant services in AWS. Our microservice architecture decouples engineering teams from each other, allowing them to build, test and deploy their services as often…Microservices4 min read
Published in Netflix TechBlog·Apr 19, 2017The Evolution of Container Usage at NetflixContainers are already adding value to our proven globally available cloud platform based on Amazon EC2 virtual machines. We’ve shared pieces of Netflix’s container story in the past (video, slides), but this blog post will discuss containers at Netflix in depth. As part of this story, we will cover Titus…Docker7 min read
Published in Netflix TechBlog·Apr 17, 2017Introducing Bolt: On Instance Diagnostic and Remediation PlatformLast August we introduced Winston, our event driven diagnostic and remediation platform. Winston helps orchestrate diagnostic and remediation actions from the outside. As part of that orchestration, there are multiple actions that need to be performed at an AWS instance(vm) level to collect data or take mitigation steps. We would…Cloud Computing10 min read
Published in Netflix TechBlog·Apr 11, 2017BetterTLSA Name Constraints test suite for HTTPS clients, created by Netflix — Written by Ian Haken At Netflix we run a microservices architecture that has hundreds of independent applications running throughout our ecosystem. One of our goals, in the interest of implementing security in depth, is to have end-to-end encrypted, authenticated communication between all of our services wherever possible, regardless of whether or not it travels…Ssl6 min read
Published in Netflix TechBlog·Apr 6, 2017Netflix Conductor: Inversion of Control for workflowsIn December 2016, we open sourced Netflix Conductor. We have been working hard since then to add more features, enhance the user interface for monitoring the workflows, and harden the system. We have seen an increase in usage, new use cases, feature requests, and lot of interest from the community…Netflix5 min read
Published in Netflix TechBlog·Mar 30, 2017The Netflix HERMES Test: Quality Subtitling at ScaleSince Netflix launched globally, the scale of our localization efforts has increased dramatically. It’s hard to believe that just 5 years ago, we only supported English, Spanish and Portuguese. Now we’ve surpassed 20 languages — including languages like Korean, Chinese, Arabic and Polish — and that number continues to grow…5 min read
Published in Netflix TechBlog·Mar 22, 2017Update on HTML5 Video for NetflixAbout four years ago, we shared our plans for playing premium video in HTML5, replacing Silverlight and eliminating the extra step of installing and updating browser plug-ins. Since then, we have launched HTML5 video on Chrome OS, Chrome, Internet Explorer, Safari, Opera, Firefox, and Edge on all supported operating systems…Technology2 min read
Published in Netflix TechBlog·Mar 14, 2017Netflix Security Monkey on Google Cloud Platform (GCP)Today we are happy to announce that Netflix Security Monkey has BETA support for tracking Google Cloud Platform (GCP) services. Initially we are providing support for the following GCP services: Firewall Rules Networking Google Cloud Storage Buckets (GCS) Service Accounts (IAM) This work was performed by a few incredible Googlers…Cloud Computing3 min read
Published in Netflix TechBlog·Mar 8, 2017Netflix Downloads on Androidby Greg Benson, Francois Goldfain, and Ashish Gupta Netflix is now a global company, so we wanted to provide a viewing experience that was truly available everywhere even when the Internet is not working well. This led to these three prioritized download use cases: Better, uninterrupted videos on unreliable Internet …Android8 min read
Published in Netflix TechBlog·Feb 21, 2017Introducing Netflix StethoscopeNetflix is pleased to announce the open source release of Stethoscope, our first project following a User Focused Security approach. The notion of “User Focused Security” acknowledges that attacks against corporate users (e.g., phishing, malware) are the primary mechanism leading to security incidents and data breaches, and it’s one of…Security4 min read
Published in Netflix TechBlog·Feb 7, 2017Introducing HubCommanderby Mike Grima, Andrew Spyker, and Jason Chan Netflix is pleased to announce the open source release of HubCommander, a ChatOps tool for GitHub management. Netflix/hubcommander hubcommander - A Slack bot for GitHub organization managementgithub.com Why HubCommander? Netflix uses GitHub, a source code management and collaboration site, extensively for both open source and internal projects. The security model for GitHub does not…Github6 min read
Published in Netflix TechBlog·Jan 30, 2017Netflix Hack Day — Winter 2017by Daniel Jacobson, Ruslan Meshenberg, Leslie Posada, and Tom Richards We hosted another great Hack Day event a week ago at Netflix headquarters. …Hackathons2 min read
Published in Netflix TechBlog·Jan 12, 2017Crafting a high-performance TV user interface using Reactby Ian McKay The Netflix TV interface is constantly evolving as we strive to figure out the best experience for our members. For example, after A/B testing, eye-tracking research, and customer feedback we recently rolled out video previews to help members make better decisions about what to watch. We’ve written…React9 min read
Published in Netflix TechBlog·Jan 3, 2017Netflix Now Supports Ultra HD 4K on Windows 10 with New Intel Core ProcessorsWe’re excited to bring Netflix support for Ultra HD 4K to Windows 10, making the vast catalog of Netflix TV shows and movies in 4K even more accessible for our members around the world to watch in the best picture quality. For the last several years, we’ve been working with…Ultra Hd 4 K2 min read
Published in Netflix TechBlog·Dec 12, 2016Netflix Conductor: A microservices orchestratorThe Netflix Content Platform Engineering team runs a number of business processes which are driven by asynchronous orchestration of tasks executing on microservices. Some of these are long running processes spanning several days. …Dynomite7 min read
Published in Netflix TechBlog·Dec 5, 2016NetflixOSS: Announcing Hollow“If you can cache everything in a very efficient way, you can often change the game” We software engineers often face problems that require the dissemination of a dataset which doesn’t fit the label “big data”. Examples of this type of problem include: Product metadata on an ecommerce site Document…Netflixoss7 min read
Published in Netflix TechBlog·Dec 1, 2016More Efficient Mobile Encodes for Netflix DownloadsLast January, Netflix launched globally, reaching many new members in 130 countries around the world. In many of these countries, people access the internet primarily using cellular networks or still-developing broadband infrastructure. Although we have made strides in delivering the same or better video quality with less bits (for example…Downloads5 min read
Published in Netflix TechBlog·Nov 22, 2016Netflix at AWS re:Invent 2016by Jason Chan Like many of our tech blog readers, Netflix is getting ready for AWS re:Invent in Las Vegas next week. Lots of Netflix engineers and recruiters will be in attendance, and we’re looking forward to meeting and reconnecting with cloud enthusiasts and Netflix OSS users. To make it…Cloud Computing6 min read
Published in Netflix TechBlog·Oct 28, 2016Netflix at RecSys 2016 — Recapby Justin Basilico and Yves Raimond A key aspect of Netflix is providing our members with a personalized experience so they can easily find great stories to enjoy. A collection of recommender systems drive the main aspects of this personalized experience and we continuously work on researching and testing new…Machine Learning2 min read
Published in Netflix TechBlog·Oct 19, 2016Netflix Chaos Monkey UpgradedWe are pleased to announce a significant upgrade to one of our more popular OSS projects. Chaos Monkey 2.0 is now on github! Years ago, we decided to improve the resiliency of our microservice architecture. At our scale it is guaranteed that servers on our cloud platform will sometimes suddenly…Chaos4 min read
Published in Netflix TechBlog·Oct 12, 2016To Be ContinuedHelping you find shows to continue watching on Netflix — by Hossein Taghavi, Ashok Chandrashekar, Linas Baltrunas, and Justin Basilico Introduction Our objective in improving the Netflix recommendation system is to create a personalized experience that makes it easier for our members to find great content to enjoy. The ultimate goal of our recommendation system is to know the exact perfect…Machine Learning10 min read
Published in Netflix TechBlog·Sep 21, 2016Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking SystemsWe recently made a major architectural change to Zuul, our cloud gateway. Did anyone even notice!? Probably not… Zuul 2 does the same thing that its predecessor did — acting as the front door to Netflix’s server infrastructure, handling traffic from all Netflix users around the world. It also routes…Zuul8 min read
Published in Netflix TechBlog·Sep 15, 2016IMF: An Open Standard with Open ToolsWhy IMF? As Netflix expanded into a global entertainment platform, our supply chain needed an efficient way to vault our masters in the cloud that didn’t require a different version for every territory in which we have our service. A few years ago we discovered the Interoperable Master Format (IMF), a standard…Open Source4 min read
Published in Netflix TechBlog·Sep 14, 2016Netflix OSS Meetup Recap — September 2016Last week, we welcomed roughly 200 attendees to Netflix HQ in Los Gatos for Season 4, Episode 3 of our Netflix OSS Meetup. The meetup group was created in 2013 to discuss our various OSS projects amongst the broader community of OSS enthusiasts. …Msl3 min read
Published in Netflix TechBlog·Sep 1, 2016Netflix Data Benchmark: Benchmarking Cloud Data StoresThe Netflix member experience is offered to 83+ million global members, and delivered using thousands of microservices. These services are owned by multiple teams, each having their own build and release lifecycles, generating a variety of data that is stored in different types of data store systems. The Cloud Database…Benchmark8 min read
Published in Netflix TechBlog·Aug 29, 2016A Large-Scale Comparison of x264, x265, and libvpx — a Sneak Peekby Jan De Cock, Aditya Mavlankar, Anush Moorthy, and Anne Aaron With 83+ million members watching billions of hours of TV shows and movies, Netflix sends a huge amount of video bits through the Internet. As we grow globally, more of these video bits will be streamed through bandwidth-constrained cellular…Video3 min read
Published in Netflix TechBlog·Aug 23, 2016Engineering Trade-Offs and The Netflix API Re-ArchitectureNetflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility. Within that framework they are free to operate with freedom to satisfy their mission. Accordingly, teams are generally responsible for all aspects of their systems, ranging…Microservices8 min read
Published in Netflix TechBlog·Aug 16, 2016Distributed delay queues based on DynomiteNetflix’s Content Platform Engineering runs a number of business processes which are driven by asynchronous orchestration of micro-services based tasks, and queues form an integral part of the orchestration layer amongst these services. Few examples of these processes are: IMF based content ingest from our partners Process of setting up…Distributed7 min read
Published in Netflix TechBlog·Aug 11, 2016Netflix and Fillby Michael Costello and Ellen Livengood — Tomorrow we’ll release another much-anticipated new series, The Get Down. Before you can hit “Play”, we have to distribute this new title to our global network of thousands of Open Connect appliances. …Content Delivery7 min read
Published in Netflix TechBlog·Aug 10, 2016Automated testing on devicesAs part of the Netflix SDK team, our responsibility is to ensure the new release version of the Netflix application is thoroughly tested to its highest operational quality before deploying onto gaming consoles and distributing as an SDK (along with a reference application) to Netflix device partners; eventually making its…Testing14 min read
Published in Netflix TechBlog·Aug 9, 2016Building fast.comOn our company blog in May, we introduced fast.com, our new internet speed test. The idea behind fast.com is to provide a quick and simple way for any internet user to test their current internet speed, whether they are a Netflix member or not. Since fast.com was released, millions of…Network Speed7 min read
Published in Netflix TechBlog·Aug 8, 2016Protecting Netflix Viewing Privacy at ScaleOn the Open Connect team at Netflix, we are always working to enhance the hardware and software in the purpose-built Open Connect Appliances (OCAs) that store and serve Netflix video content. As we mentioned in a recent company blog post, since the beginning of the Open Connect program we have…Aes Gcm4 min read
Published in Netflix TechBlog·Aug 5, 2016Introducing Winston — Event driven Diagnostic and Remediation PlatformNetflix is a collection of microservices that all come together to enable the product you have come to love. Operating these micro services is also distributed across the owning teams and their engineers. We do not run a central operations team managing these individual services for availability. What we do…Automation9 min read
Published in Netflix TechBlog·Aug 3, 2016Vizceral Open SourcePreviously we wrote about our traffic intuition tool, Flux. We have some announcements and updates to share about this project. First, we have renamed the project to Vizceral. More importantly, Vizceral is now open source! Flux: A New Approach to System Intuition a holistic understanding of traffic as it flows through our complex, distributed systemmedium.com Open Source Vizceral transformed the way we understand and digest information about the state of traffic flowing…Cloud6 min read
Published in Netflix TechBlog·Aug 1, 2016Netflix Billing Migration to AWS — Part IIIIn the billing migration blog post published a few weeks ago, we explained the overall approach employed in migrating our billing system to the cloud: Netflix Billing Migration to AWS part of a broader initiative towards AWS migrationmedium.com In this post, the database migration portion will be covered in detail. …Cloud Architecture10 min read
Published in Netflix TechBlog·Jul 29, 2016Distributed Resource Scheduling with Apache MesosNetflix uses Apache Mesos to run a mix of batch, stream processing, and service style workloads. For over two years, we have seen an increased usage for a variety of use cases including real time anomaly detection, training and model building batch jobs, machine learning orchestration, and Node.js based microservices…Apache Mesos6 min read
Published in Netflix TechBlog·Jul 20, 2016Netflix Billing Migration to AWS — Part IIThis is a continuation in the series on Netflix Billing migration to the Cloud. An overview of the migration project was published earlier here: Netflix Billing Migration to AWS part of a broader initiative towards AWS migrationmedium.com This post details the technical journey for the Billing applications and datastores as they were moved from the Data Center to AWS Cloud. As you might…Billing12 min read
Published in Netflix TechBlog·Jul 18, 2016Chelsea: Encoding in the Fast LaneBack in May Netflix launched its first global talk show: Chelsea. Delivering this new format was a first for us, and a fun challenge in many different aspects, which this blog describes in more detail: Delivering a Global Talk Show: The Innovation Behind Chelsea Delivering a Global Talk Show: The Innovation Behind Chelsea Last week, we launched a talk show on Netflix -- a first…media.netflix.com Chelsea Handler’s new Netflix talk show ushered in a Day-of-Broadcast (DOB) style of delivery that…Encoding9 min read
Published in Netflix TechBlog·Jul 12, 2016Global Languages Support at NetflixTesting Search Queries — By Ivan Provalov Globalization at Netflix Having launched the Netflix service globally in January, we now support search in 190 countries. We currently support 20 languages, and this will continue to grow over time. Some of the most challenging language support was added while launching in Japan and Korea as well as in…Encoding9 min read
Published in Netflix TechBlog·Jul 5, 2016Product Integration Testing at the Speed of NetflixThe Netflix member experience is delivered using a micro-service architecture and is personalized to each of our 80+ million members. These services are owned by multiple teams, each having their own lifecycle of build and release. …Testing9 min read
Published in Netflix TechBlog·Jun 27, 2016Netflix and the IMF CommunityPhoton OSS — IMF validation for the masses — When you’ve got something this good, it’s hard to keep it to yourself. As we develop our IMF validation tools internally, we are merging them into a public git repository. The project, code-named Photon (to imply a torchbearer), was envisioned to aid a wider adoption of the IMF standard, and…Imf9 min read
Published in Netflix TechBlog·Jun 21, 2016Netflix Billing Migration to AWSOn January 4, 2016, right before Netflix expanded itself into 130 new countries, Netflix Billing infrastructure became 100% AWS cloud-native. Migration of Billing infrastructure from Netflix Data Center(DC) to AWS Cloud was part of a broader initiative. …Billing9 min read
Published in Netflix TechBlog·Jun 6, 2016Toward A Practical Perceptual Video Quality Metricby Zhi Li, Anne Aaron, Ioannis Katsavounidis, Anush Moorthy and Megha Manohara At Netflix we care about video quality, and we care about measuring video quality accurately at scale. Our method, Video Multimethod Assessment Fusion (VMAF), seeks to reflect the viewer’s perception of our streaming quality. We are open-sourcing this…Encoding19 min read
Published in Netflix TechBlog·Jun 1, 2016Dynomite-manager: Managing Dynomite ClustersDynomite has been adopted widely inside Netflix due to its high performance and low latency attributes. In our recent blog, we showcased the performance of Dynomite with Redis as the underlying data storage engine: Dynomite with Redis on AWS — Benchmarks performance results using AWS with and without the recently added consistency featuremedium.com At this point (Q2 2016), there are almost 50 clusters with more than 1000 nodes, centrally…Database9 min read
Published in Netflix TechBlog·May 31, 2016Meson: Workflow Orchestration for Netflix RecommendationsAt Netflix, our goal is to predict what you want to watch before you watch it. To do this, we run a large number of machine learning (ML) workflows every day. In order to support the creation of these workflows and make efficient use of resources, we created Meson. Meson…Infrastructure8 min read
Published in Netflix TechBlog·May 25, 2016Application data caching using SSDsThe Moneta project: Next generation EVCache for better cost optimization — With the global expansion of Netflix earlier this year came the global expansion of data. After the Active-Active project and now with the N+1 architecture, the latest personalization data needs to be everywhere at all times to serve any member from any region. …Caching13 min read
Published in Netflix TechBlog·May 24, 2016Netflix Hack Day — Spring 2016by Daniel Jacobson, Ruslan Meshenberg, and Leslie Posada Keeping with our roughly six-month cadence, Netflix recently hosted another fantastic Hack Day event. …Hackathons2 min read
Published in Netflix TechBlog·May 11, 2016Building High-Performance Mobile Applications at NetflixOn April 21, we hosted our first Mobile event at Netflix with two talks focusing on the challenges we’ve faced building performant mobile experiences, and how we addressed them. Francois Goldfain, Engineering Manager for the Android Platform team, kicked off the event with his talk on optimizing the Netflix Android…Android1 min read
Published in Netflix TechBlog·May 5, 2016Highlights from PRS2016 workshopPersonalized recommendations and search are the primary ways Netflix members find great content to watch. We’ve written much about how we build them and some of the open challenges. Recently we organized a full-day workshop at Netflix on Personalization, Recommendation, and Search (PRS2016), bringing together researchers and practitioners in these…Workshopprs5 min read
Published in Netflix TechBlog·May 3, 2016Selecting the best artwork for videos through A/B testingAt Netflix, we are constantly looking at ways to help our 81.5M members discover great stories that they will love. A big part of that is creating a user experience that is intuitive, fun, and meaningfully helps members find and enjoy stories on Netflix as fast as possible. This blog…Ab Testing10 min read
Published in Netflix TechBlog·Apr 29, 2016It’s All A/Bout Testing: The Netflix Experimentation PlatformEver wonder how Netflix serves a great streaming experience with high-quality video and minimal playback interruptions? Thank the team of engineers and data scientists who constantly A/B test their innovations to our adaptive streaming and content delivery network algorithms. What about more obvious changes, such as the complete redesign of…Ab Testing11 min read
Published in Netflix TechBlog·Apr 27, 2016Kafka Inside Keystone PipelineThis is the second blog of our Keystone pipeline series. Please refer to the first part for overview and evolution of the Keystone pipeline: Evolution of the Netflix Data Pipeline our Keystone data pipeline has experienced major transformationsmedium.com In summary, the Keystone pipeline is a unified event publishing, collection, and routing infrastructure for both batch and stream processing. We have two sets of Kafka…Keystone8 min read
Published in Netflix TechBlog·Apr 18, 2016A Scalable System for Ingestion and Delivery of Timed TextOffering the same great Netflix experience to diverse audiences and cultures around the world is a core aspect of the global Netflix video delivery service. With high quality subtitle localization being a key component of the experience, we have developed (and are continuously refining) a Unicode standard based i18n-grade timed…Imf11 min read
Published in Netflix TechBlog·Apr 11, 2016Saving 13 Million Computational Minutes per Day with Flame GraphsWe recently uncovered a significant performance win for a key microservice here at Netflix, and we want to share the problems, challenges, and new developments for the kind of analysis that lead us to the optimization. Optimizing performance is critical for Netflix. Improvements enhance the customer experience by lowering latency…Efficiency8 min read
Published in Netflix TechBlog·Apr 4, 2016The Netflix IMF WorkflowFollowing our introductory article, this (the second in our IMF series) post describes the Netflix IMF ingest implementation and how it fits within the scope of our content processing pipeline. While conventional essence containers (e.g., QuickTime) commonly include essence data in the container file, the IMF CPL is designed to…Imf6 min read
Published in Netflix TechBlog·Apr 1, 2016How Netflix Uses John Stamos to Optimize the Cloud at ScaleNetflix Technology relies heavily on the Cloud, thanks to its low latency and high compatibility with the internet. But the key to great Technology is having great Talent. …Netflix3 min read
Published in Netflix TechBlog·Mar 30, 2016Global Cloud — Active-Active and BeyondThis is a continuing post on the Netflix architecture for Global Availability. In the past we talked about efforts like Isthmus and Active-Active: Isthmus — Resiliency against ELB outages achieving multi-regional ELB resiliencymedium.com Active-Active for Multi-Regional Resiliency With large scale and velocity there is increased chance of failure.medium.com We continue the story from where we left off at the end of the Active-Active project in 2013. We had achieved multi-regional resiliency for our members in…AWS7 min read
Published in Netflix TechBlog·Mar 23, 2016Performance without CompromiseLast week we hosted our latest Netflix JavaScript Talks event at our headquarters in Los Gatos, CA. We gave two talks about our unflinching stance on performance. In our first talk, Steve McGuire shared how we achieved a completely declarative, React-based architecture that’s fast on the devices in your living…Java Script1 min read
Published in Netflix TechBlog·Mar 21, 2016Extracting image metadata at scaleWe have a collection of nearly two million images that play very prominent roles in helping members pick what to watch. This blog describes how we use computer vision algorithms to address the challenges of focal point, text placement and image clustering at a large scale. Focal point All images have a…Computer Vision5 min read
Published in Netflix TechBlog·Mar 14, 2016Stream-processing with MantisBack in January of 2014 we wrote about the need for better visibility into our complex operational environments: Improving Netflix’s Operational Visibility with Real-Time Insight Tools exceptional tools coupled with intelligent analysismedium.com The core of the message in that post was about the need for fine-grained, contextual and scalable insights into the experiences of our customers and behaviors of our services. While our execution…Real Time Insights9 min read
Published in Netflix TechBlog·Mar 9, 2016How We Build Code at NetflixHow does Netflix build code before it’s deployed to the cloud? While pieces of this story have been told in the past, we decided it was time we shared more details. …Continuous Integration8 min read
Published in Netflix TechBlog·Mar 7, 2016IMF: A Prescription for VersionitisThis blog post provides an introduction to the emerging IMF (Interoperable Master Format) standard from SMPTE (The Society of Motion Picture and Television Engineers), and delves into a short case study that highlights some of the operational benefits that Netflix receives from IMF today. Have you ever noticed that your…Imf7 min read
Published in Netflix TechBlog·Mar 1, 2016Caching for a Global Netflix#CachesEverywhere — Netflix members have come to expect a great user experience when interacting with our service. …Programming12 min read
Published in Netflix TechBlog·Feb 17, 2016Recommending for the World#AlgorithmsEverywhere — by Yves Raimond and Justin Basilico The Netflix experience is driven by a number of Machine Learning algorithms: personalized ranking, page generation, search, similarity, ratings, etc. On the 6th of January, we simultaneously launched Netflix in 130 new countries around the world, which brings the total to over 190 countries…Algorithms9 min read
Published in Netflix TechBlog·Feb 15, 2016Evolution of the Netflix Data PipelineOur new Keystone data pipeline went live in December of 2015. In this article, we talk about the evolution of Netflix’s data pipeline over the years. This is the first of a series of articles about the new Keystone data pipeline. Netflix is a data-driven company. Many business and product…Chukwa4 min read
Published in Netflix TechBlog·Feb 12, 2016Distributed Time Travel for Feature GenerationWe want to make it easy for Netflix members to find great content to fulfill their unique tastes. To do this, we follow a data-driven algorithmic approach based on machine learning, which we have described in past posts and other publications. We aspire to a day when anyone can sit…Machine Learning15 min read
Published in Netflix TechBlog·Feb 1, 2016Astyanax — Retiring an old friendIn the summer of 2011, Astyanax, an Apache Cassandra (C*) Java client library was created to easily consume Cassandra, which at the time was in its infancy. Astyanax became so popular that for a good while, it became the de facto java client library for the Apache Cassandra community. …Cassandra2 min read
Published in Netflix TechBlog·Jan 20, 2016Automated Failure Testinga.k.a. Training Smarter Monkeys — At Netflix, we have found that proactive failure testing is a great way to ensure that we have a reliable product for our members by helping us prepare our systems, and our teams, for the problems that arise in our production environment. Our various efforts in this space, some of…Testing7 min read
Published in Netflix TechBlog·Jan 14, 2016Dynomite with Redis on AWS — BenchmarksAbout a year ago the Cloud Database Engineering (CDE) team published a post Introducing Dynomite. Introducing Dynomite Making Non-Distributed Databases Distributedmedium.com Dynomite is a proxy layer that provides sharding and replication and can turn existing non-distributed datastores into a fully distributed system with multi-region replication. One of its core features is the ability to scale a…Database9 min read
Published in Netflix TechBlog·Dec 17, 2015HTML5 Video is now supported in FirefoxToday we’re excited to announce the availability of our HTML5 player in Firefox! Windows support is rolling out this week, and OS X support will roll out next year. Firefox ships with the very latest versions of the HTML5 Premium Video Extensions. That includes the Media Source Extensions (MSE), which…Html 52 min read
Published in Netflix TechBlog·Dec 14, 2015Per-Title Encode OptimizationWe’ve spent years developing an approach, called per-title encoding, where we run analysis on an individual title to determine the optimal encoding recipe based on its complexity. Imagine having very involved action scenes that need more bits to encapsulate the information versus unchanging landscape scenes or animation that need less…Video11 min read
Published in Netflix TechBlog·Dec 10, 2015Optimizing Content Quality Control at Netflix with Predictive Modelingby Nirmal Govind and Athula Balachandran Over 69 million Netflix members stream billions of hours of movies and shows every month in North and South America, parts of Europe and Asia, Australia and New Zealand. …Machine Learning5 min read
Published in Netflix TechBlog·Dec 9, 2015High Quality Video Encoding at ScaleAt Netflix we receive high quality sources for our movies and TV shows and encode them to the best video streams possible for a given member’s viewing device and bandwidth capabilities. With the continued growth of our service it has been essential to build a video encoding pipeline that is…Video7 min read
Published in Netflix TechBlog·Dec 3, 2015Debugging Node.js in Productionby Kim Trott and Yunong Xiao We recently hosted our latest JavaScript Talks event on our new campus at Netflix headquarters in Los Gatos, California. Yunong Xiao, senior software engineer on our Node.js platform, presented on debugging Node.js in production. Yunong showed hands-on techniques using the scientific method to root cause and solve for runtime performance issues, crashes, errors, and memory leaks.Nodejs1 min read
Published in Netflix TechBlog·Dec 1, 2015Caching Content for Holiday Streaming’Tis the season for holiday binging. How do seasonal viewing patterns affect how Netflix stores and streams content? Our solution for delivering streaming content is Open Connect, a custom-built system that distributes and stores the audio and video content our members download when they stream a Netflix title (a movie…Analytics3 min read
Published in Netflix TechBlog·Nov 30, 2015Linux Performance Analysis in 60,000 MillisecondsYou log in to a Linux server with a performance issue: what do you check in the first minute? At Netflix we have a massive EC2 Linux cloud, and numerous performance analysis tools to monitor and investigate its performance. These include Atlas for cloud-wide monitoring, and Vector for on-demand instance…Linux11 min read
Published in Netflix TechBlog·Nov 23, 2015Creating Your Own EC2 Spot Market — Part 2In Part 1 Creating Your Own EC2 Spot Market of this series, we explained how Netflix manages its EC2 footprint and how we take advantage of our daily peak of 12,000 unused instances which we named the “internal spot market.” Creating Your Own EC2 Spot Market incorporating hybrid autoscaling engines for our servicesmedium.com This sizeable trough has significantly improved our encoding throughput, and…AWS8 min read
Published in Netflix TechBlog·Nov 20, 2015Sleepy Puppy Extension for Burp SuiteNetflix recently open sourced Sleepy Puppy — a cross-site scripting (XSS) payload management framework for security assessments. Announcing Sleepy Puppy Cross-Site Scripting Payload Management for Web Application Security Testingmedium.com One of the most frequently requested features for Sleepy Puppy has been for an extension for Burp Suite, an integrated platform for web application security testing. Today, we are pleased to open source…Security3 min read
Published in Netflix TechBlog·Nov 16, 2015Global Continuous Delivery with SpinnakerAfter over a year of development and production use at Netflix, we’re excited to announce that our Continuous Delivery platform, Spinnaker, is available on GitHub. Spinnaker is an open source multi-cloud Continuous Delivery platform for releasing software changes with high velocity and confidence. Spinnaker is designed with pluggability in mind…Continuous Delivery3 min read
Published in Netflix TechBlog·Nov 9, 2015Netflix Hack Day — Autumn 2015by Daniel Jacobson, Ruslan Meshenberg, Matt McCarthy, and Leslie Posada Last week, we hosted our latest installment of Netflix Hack Day. As always, Hack Day is a way for our product development staff to get away from everyday work, to have fun, experiment, collaborate, and be creative. The following video…Hackathons3 min read
Published in Netflix TechBlog·Oct 28, 2015Evolution of Open Source at NetflixWhen we started our Netflix Open Source (aka NetflixOSS) Program several years ago, we didn’t know how it would turn out. We did not know whether our OSS contributions would be used, improved, or ignored; whether we’d have a community of companies and developers sending us feedback; and whether middle-tier…Open Source4 min read
Published in Netflix TechBlog·Oct 20, 2015Falcor for AndroidWe’re happy to have open-sourced the Netflix Falcor library earlier this year. On Android, we wanted to make use of Falcor in our client app for its efficient model of data fetching as well as its inherent cache coherence. Falcor requires us to model data on both the client and…Android7 min read
Published in Netflix TechBlog·Oct 2, 2015Netflix at AWS re:Invent 2015Ever since AWS started the re:Invent conference, Netflix has actively participated each and every year. This year is no exception, and we’re planning on presenting at 8 different sessions. …Cloud Computing6 min read
Published in Netflix TechBlog·Oct 1, 2015Flux: A New Approach to System IntuitionOn the Traffic and Chaos Teams at Netflix, our mission requires that we have a holistic understanding of our complex microservice architecture. At any given time, we may be called upon to move the request traffic of many millions of customers from one side of the planet to the other…Microservices6 min read
Published in Netflix TechBlog·Sep 30, 2015Moving from Asgard to SpinnakerSix years ago, Netflix successfully jumped headfirst into the AWS Cloud and along the way we ended up writing quite a lot of software to help us out. One particular project proved instrumental in allowing us to efficiently automate AWS deployments: Asgard. Asgard: Web-based Cloud Management and Deployment Asgard, a web interface for application deployments and cloud management.medium.com Asgard created an intuitive model for cloud-based applications…AWS4 min read
Published in Netflix TechBlog·Sep 28, 2015Creating Your Own EC2 Spot Marketby Andrew Park, Darrell Denlinger, & Coburn Watson Netflix prioritizes innovation and reliability above efficiency, and as we continue to scale globally, finding opportunities that balance these three variables becomes increasingly difficult. …AWS4 min read
Published in Netflix TechBlog·Sep 25, 2015Chaos Engineering UpgradedSeveral years ago we introduced a tool called Chaos Monkey. This service pseudo-randomly plucks a server from our production deployment on AWS and kills it. At the time we were met with incredulity and skepticism. Are we crazy? In production?!? Our reasoning was sound, and the results bore that out…Chaos Engineering6 min read
Published in Netflix TechBlog·Sep 24, 2015John Carmack on Developing the Netflix App for OculusHi, this is Anthony Park, VP of Engineering at Netflix. We’ve been working with Oculus to develop a Netflix app for Samsung Gear VR. The app includes a Netflix Living Room, allowing members to get the Netflix experience from the comfort of a virtual couch, wherever they bring their Gear…Virtual Reality11 min read
Published in Netflix TechBlog·Sep 22, 2015Announcing Electric Eyeby Michael Russell Netflix ships on a wide variety of devices, ranging from small thumbdrive-sized HDMI dongles to ultra-massive 100”+ curved screen HDTVs, and the wide variety of form factors leads to some interesting challenges in testing. …Appsanddevices7 min read
Published in Netflix TechBlog·Sep 21, 2015Introducing Lemurby Kevin Glisson, Jason Chan, and Ben Hagen Netflix is pleased to announce the open source release of our x.509 certificate orchestration framework: Lemur! The Challenge of Certificate Management Public Key Infrastructure is a set of hardware, software, people, policies, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates and manage…Security7 min read
Published in Netflix TechBlog·Aug 31, 2015Announcing Sleepy Puppy — Cross-Site Scripting Payload Management for Web Application Security Testingby Scott Behrens and Patrick Kelley Netflix is pleased to announce the open source release of our cross-site scripting (XSS) payload management framework: Sleepy Puppy! The Challenge of Cross-Site Scripting Cross-site scripting is a type of web application security vulnerability that allows an attacker to execute arbitrary client-side script in a victim’s browser. XSS has…Security6 min read
Published in Netflix TechBlog·Aug 25, 2015From Chaos to Control — Testing the resiliency of Netflix’s Content Discovery Platformby Leena Janardanan, Bruce Wobbe, Vilas Veeraraghavan Introduction Merchandising Application Platform (MAP) was conceived as a middle-tier service that would handle real time requests for content discovery. MAP does this by aggregating data from disparate data sources and implementing common business logic into one distinct layer. This centralized layer helps provide…Resilience7 min read
Published in Netflix TechBlog·Aug 20, 2015Fenzo: OSS Scheduler for Apache Mesos FrameworksBringing Netflix to our millions of subscribers is no easy task. The product comprises dozens of services in our distributed environment, each of which is operating a critical component to the experience while constantly evolving with new functionality. Optimizing the launch of these services is essential for both the stability…Fenzo6 min read
Published in Netflix TechBlog·Aug 17, 2015Netflix Releases Falcor Developer Previewby Jafar Husain, Paul Taylor, and Michael Paulson Developers strive to create the illusion that all of their application’s data is sitting right there on the user’s device just waiting to be displayed. …Java Script2 min read
Published in Netflix TechBlog·Aug 5, 2015Making Netflix.com Fasterby Kristofer Baxter Simply put, performance matters. We know members want to immediately start browsing or watching their favorite content and have found that faster startup leads to more satisfying usage. So, when building the long-awaited update to netflix.com, …Java Script4 min read
Published in Netflix TechBlog·Aug 3, 2015Netflix at Velocity 2015: Linux Performance ToolsThere are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? At Velocity 2015, I gave a 90 minute tutorial on Linux performance tools. I’ve spoken on this topic before, but given a 90 minute time slot I was able…Linux2 min read
Published in Netflix TechBlog·Jul 28, 2015Tuning Tomcat For A High Throughput, Fail Fast Systemby Sumit Tandon Problem Netflix has a number of high throughput, low latency mid tier services. In one of these services, it was observed that in case there is a huge surge in traffic in a very short span of time, the machines became cpu-starved and would become unresponsive. This would…Tomcat8 min read
Published in Netflix TechBlog·Jul 24, 2015Java in FlamesJava mixed-mode flame graphs provide a complete visualization of CPU usage and have just been made possible by a new JDK option: -XX:+PreserveFramePointer. We’ve been developing these at Netflix for everyday Java performance analysis as they can identify all CPU consumers and issues, including those that are hidden from other…Java12 min read
Published in Netflix TechBlog·Jul 14, 2015Tracking down the Villains: Outlier Detection at NetflixIt’s 2 a.m. and half of our reliability team is online searching for the root cause of why Netflix streaming isn’t working. None of our systems are obviously broken, but something is amiss and we’re not seeing it. After an hour of searching we realize there is one rogue server…Machine Learning7 min read
Published in Netflix TechBlog·Jun 15, 2015NTS: Real-time Streaming for Test Automationby Peter Hausel and Jwalant Shah Netflix Test Studio Netflix members can enjoy instant access to TV shows & Movies on over 1400 different device/OS permutations. Assessing long-duration playback quality and delivering a great member experience on such a diverse set of playback devices presented a huge challenge to the team. Netflix Test…Appsanddevices5 min read
Published in Netflix TechBlog·Jun 4, 2015Localization Technologies at NetflixThe localization program at Netflix is centered around linguistic excellence, a great team environment, and cutting-edge technology. The program is only 4 years old, which for a company our size is unusual to find. …Localization7 min read
Published in Netflix TechBlog·May 27, 2015Netflix Streaming — More Energy Efficient than BreathingNetflix Streaming: Energy Consumption for 2014 was 0.0013 kWh per Streaming Hour Delivered. 36% was from renewable sources 28% was offset with renewable energy credits We plan to be fully offset by 2015, and to increase the contribution of renewable sources Carbon footprint of about 300g of CO2 per customer…Energy5 min read
Published in Netflix TechBlog·May 4, 2015Introducing FIDO: Automated Security Incident ResponseWe’re excited to announce the open source release of FIDO (Fully Integrated Defense Operation — apologies to the FIDO Alliance for acronym collision), our system for automatically analyzing security events and responding to security incidents. Overview The typical process for investigating security-related alerts is labor intensive and largely manual. To make…Security5 min read
Published in Netflix TechBlog·Apr 9, 2015Learning a Personalized Homepageby Chris Alvino and Justin Basilico As we’ve described in our previous blog posts, at Netflix we use personalization extensively and treat every situation as an opportunity to present the right content to each of our over 57 million members. The main way a member interacts with our recommendations is…Machine Learning15 min read
Published in Netflix TechBlog·Apr 8, 2015Introducing Vector: Netflix’s On-Host Performance Monitoring Toolby Martin Spier, Amer Ather, and Brendan Gregg Vector is an open source host-level performance monitoring framework, which exposes hand-picked, high-resolution system and application metrics to every engineer’s browser. Having the right metrics available on demand and at a high resolution is key to understanding how a system behaves and…Vector6 min read
Published in Netflix TechBlog·Apr 6, 2015Extracting contextual information from video assetsby Apurva Kansara Here, I will describe our approach to extract contextual metadata from video assets to enable an improved Netflix user experience across the large catalog we serve. Part 1: Detecting End-Sequences When you finish watching a movie, we are able to provide a unique post-play experience as illustrated below in two examples…Text Recognition3 min read
Published in Netflix TechBlog·Mar 30, 2015Billing & Payments Engineering Meetup IIOn March 18th, we hosted our second Billing & Payments Engineering Meetup at Netflix. It felt truly encouraging to see the growing interest of the engineer community of the Bay Area for the event. Just like the first event, the theater was almost full. If you missed our first Meetup…Payments3 min read
Published in Netflix TechBlog·Mar 11, 2015Can Spark Streaming survive Chaos Monkey?by Bharat Venkat, Prasanna Padmanabhan, Antony Arokiasamy, and Raju Uppalapati Netflix is a data-driven organization that places emphasis on the quality of data collected and processed. In our previous blog post, we highlighted our use cases for real-time stream processing in the context of online recommendations and data monitoring: What’s trending on Netflix? improving our recommender systemsmedium.comApache Spark4 min read
Published in Netflix TechBlog·Mar 9, 2015Netflix Hack Day — Winter 2015by Daniel Jacobson, Ruslan Meshenberg, Matt McCarthy, and Leslie Posada Last week, we hosted the latest Netflix Hack Day. Hack Day is a way for our product development teams to get away from everyday work. It’s a fun, experimental, collaborative, and creative outlet. This time, we had about 70 hacks…Hackathons3 min read
Published in Netflix TechBlog·Feb 19, 2015RAD — Outlier Detection on Big DataOutlier detection can be a pain point for all data driven companies, especially as data volumes grow. At Netflix we have multiple datasets growing by 10B+ record/day and so there’s a need for automated anomaly detection tools ensuring data quality and identifying suspicious anomalies. …Data Science3 min read
Published in Netflix TechBlog·Feb 18, 2015A Microscope on Microservicesby Coburn Watson, Scott Emmons, and Brendan Gregg At Netflix we pioneer new cloud architectures and technologies to operate at massive scale — a scale which breaks most monitoring and analysis tools. The challenge is not just handling a massive instance count but to also provide quick, actionable insight for…Observability5 min read
Published in Netflix TechBlog·Feb 10, 2015Nicobar: Dynamic Scripting Library for Javaby James Kojo, Vasanth Asokan, George Campbell, Aaron Tull The Netflix API is the front door to the streaming service, handling billions of requests per day from more than 1000 different device types around the world. To provide the best experience to our subscribers, it is critical that our UI…API6 min read
Published in Netflix TechBlog·Feb 10, 2015What’s trending on Netflix?by Prasanna Padmanabhan, Kedar Sadekar, Gopal Krishnan Every day, millions of members across the globe, from thousands of devices, visit Netflix and generate millions of viewing hours. The majority of these viewing hours are generated through the videos that are recommended by our recommender systems. We continue to invest in…Big Data5 min read
Published in Netflix TechBlog·Feb 2, 2015SPS: the Pulse of Netflix StreamingWe want to provide an amazing experience to each member, winning the “moments of truth” where they decide what entertainment to enjoy. To do that, we need to understand the health of our system. To quickly and easily understand the health of the system, we need a simple metric that…High Availability7 min read
Published in Netflix TechBlog·Jan 28, 2015Netflix Likes ReactWe are making big changes in the way we build the Netflix experience with Facebook’s React library. Today, we will share our thoughts on what makes React so compelling and how it is evolving our approach to UI development. At the beginning of last year, Netflix UI engineers embarked on…Java Script4 min read
Published in Netflix TechBlog·Jan 27, 2015Netflix’s Viewing DataHow We Know Where You Are in House of Cards — Over the past 7 years, Netflix streaming has expanded from thousands of members watching occasionally to millions of members watching over two billion hours every month. Each time a member starts to watch a movie or TV episode, a “view” is created in our data systems and a collection of…Cassandra6 min read
Published in Netflix TechBlog·Jan 20, 2015Introducing Surus and ScorePMMLToday we’re announcing a new Netflix-OSS project called Surus. Over the next year we plan to release a handful of our internal user defined functions (UDF’s) that have broad adoption across Netflix. The use cases for these functions are varied in nature (e.g. scoring predictive models, outlier detection, pattern matching…Big Data4 min read
Published in Netflix TechBlog·Dec 12, 2014Introducing Atlas: Netflix’s Primary Telemetry PlatformVarious previous Tech Blog posts have referred to our centralized monitoring system, and we’ve presented at least one talk about it previously. Today, we want to both discuss the platform and ecosystem we built for time-series telemetry and its capabilities and announce the open-sourcing of its underlying foundation. How We Got Here While working…Big Data10 min read
Published in Netflix TechBlog·Nov 19, 2014Node.js in FlamesWe’ve been busy building our next-generation Netflix.com web application using Node.js. You can learn more about our approach from the presentation we delivered at NodeConf.eu a few months ago. Today, I want to share some recent learnings from performance tuning this new application stack. We were first clued in to…Nodejs7 min read
Published in Netflix TechBlog·Nov 12, 2014ZeroToDocker: An easy way to evaluate NetflixOSS through runtime packagingMotivation: The NetflixOSS platform and related ecosystem services are extensive. While we make every attempt to document each project, being able to quickly evaluate NetflixOSS is a large challenge due to the breadth for most users. …Docker6 min read
Published in Netflix TechBlog·Nov 11, 2014Genie 2.0: Second Wish Granted!by Tom Gianos and Amit Sharma @ Big Data Platform Team A little over a year ago we announced Genie, a distributed job and resource management tool. Since then, Genie has operated in production at Netflix, servicing tens of thousands of ETL and analytics jobs daily. …Big Data7 min read
Published in Netflix TechBlog·Nov 10, 2014Introducing Raigad — An Elasticsearch SidecarNetflix has very diverse data needs. Those needs fall anywhere between rock-solid durable datastores, like Apache Cassandra and lossy in-memory stores, such as the current incarnation of Dynomite. Somewhere in that spectrum is the need to store, index, and search documents. …Elasticsearch3 min read
Published in Netflix TechBlog·Nov 5, 2014Prana: A Sidecar for your Netflix PaaS based Applications and Services“The Right Tool for the Right Job” It’s hard to argue against this time tested mantra. At Netflix, an overwhelming part of our applications and services have traditionally been implemented in Java. As our services and products evolved, we asked ourselves if Java was still the right choice for implementing…Java6 min read
Published in Netflix TechBlog·Nov 3, 2014Introducing Dynomite — Making Non-Distributed Databases, DistributedIntroduction & Overview Netflix has long been a proponent of the microservices model. This model offers higher-availability, resiliency to failure and loose coupling. The downside to such an architecture is the potential for a latent user experience. Every time a customer loads up a homepage or starts to stream a movie, there are…Database9 min read
Published in Netflix TechBlog·Oct 31, 2014Message Security Layer: A Modern Take on Securing CommunicationNetflix serves audio and video to millions of devices and subscribers across the globe. Each device has its own unique hardware and software, and differing security properties and capabilities. The communication between these devices and our servers must be secured to protect both our subscribers and our service. When we…Security7 min read
Published in Netflix TechBlog·Oct 23, 2014FIT: Failure Injection Testingby Kolton Andrus, Naresh Gopalani, Ben Schmaus It’s no secret that at Netflix we enjoy deliberately breaking things to test our production systems. Doing so lets us validate our assumptions and prove that our mechanisms for handling failure will work when called upon. …Resilience4 min read
Published in Netflix TechBlog·Oct 7, 2014Using Presto in our Big Data Platform on AWSby Eva Tse, Zhenxiao Luo, Nezih Yigitbasi @ Big Data Platform team At Netflix, the Big Data Platform team is responsible for building a reliable data analytics platform shared across the whole company. In general, Netflix product decisions are very data driven. So we play a big role in helping…Big Data9 min read
Published in Netflix TechBlog·Sep 25, 2014Inviso: Visualizing Hadoop Performanceby Daniel C. Weeks In a post last year we discussed our big data architecture and the advantages of working with big data in the cloud (read more here). One of the key points from the article is that Netflix leverages Amazon’s Simple Storage Service (S3) as the “source of…Big Data8 min read
Published in Netflix TechBlog·Aug 25, 2014Announcing Scumblr and Sketchy — Search, Screenshot, and Reclaim the InternetNetflix is pleased to announce the open source release of two security-related web applications: Scumblr and Sketchy! Scumbling The Web Many security teams need to stay on the lookout for Internet-based discussions, posts, and other bits that may be of impact to the organizations they are protecting. These teams then take a variety…Web Development4 min read
Published in Netflix TechBlog·Aug 20, 2014Netflix Hack Day — Summer 2014by Daniel Jacobson, Ruslan Meshenberg, Matt McCarthy, and Leslie Posada Hack Day is a tradition at Netflix, as it is for many Silicon Valley companies. …Hackathons3 min read
Published in Netflix TechBlog·Aug 18, 2014Scaling A/B Testing on Netflix.com with Node.jsby Chris Saint-Amant Last Wednesday night we held our third JavaScript Talks event at our Netflix headquarters in Los Gatos, Calif. Alex Liu and Micah Ransdell discussed some of the unique UI engineering challenges that go along with running hundreds of A/B tests each year We are currently migrating our…Java Script2 min read
Published in Netflix TechBlog·Jul 25, 2014Revisiting 1 Million Writes per secondby Christos Kalantzis In an article we posted in November 2011, Benchmarking Cassandra Scalability on AWS — Over a million writes per second, we showed how Cassandra (C*) scales linearly as you add more nodes to a cluster. Benchmarking Cassandra Scalability on AWS Over a million writes per secondmedium.com With the advent of new EC2 instance types, we decided to revisit…Cassandra8 min read
Published in Netflix TechBlog·Jun 30, 2014Announcing Security Monkey — AWS Security Configuration Monitoring and AnalysisWe are pleased to announce the open source availability of Security Monkey, our solution for monitoring and analyzing the security of our Amazon Web Services configurations. At Netflix, responsibility for delivering the streaming service is distributed and the environment is constantly changing. Code is deployed thousands of times a day…AWS6 min read
Published in Netflix TechBlog·Jun 16, 2014Delivering Breaking Bad on Netflix in Ultra HD 4KThis week Netflix is pleased to begin streaming all 62 episodes of Breaking Bad in UltraHD 4K. Breaking Bad in 4K comes from Sony Pictures Entertainment’s beautiful remastering of Breaking Bad from the original film negatives. This 4K experience is available on select 4K Smart TVs. As pleased as I…Sony2 min read
Published in Netflix TechBlog·Jun 11, 2014Optimizing the Netflix Streaming Experience with Data Scienceby Nirmal Govind On January 16, 2007, Netflix started rolling out a new feature: members could now stream movies directly on their browser without having to wait for the red envelope in the mail. This event marked a substantial shift for Netflix and the entertainment industry. A lot has changed…Algorithms6 min read
Published in Netflix TechBlog·Jun 3, 2014HTML5 Video in Safari on OS X Yosemiteby Anthony Park and Mark Watson. We’re excited to announce that Netflix streaming in HTML5 video is now available in Safari on OS X Yosemite! We’ve been working closely with Apple to implement the Premium Video Extensions in Safari, which allow playback of premium video content in the browser without…OSX2 min read
Published in Netflix TechBlog·May 16, 2014Improving the performance of our JavaScript inheritance modelby Dylan Oudyk When building the initial architecture for Netflix’s common TV UI in early 2009, we knew we wanted to use inheritance to share common functionality in our codebase. Since the first engineers working on the code were coming from the heavily Java-based Website, they preferred simulating a classical…Java Script5 min read
Published in Netflix TechBlog·Apr 1, 2014Women in Technology meetup at Netflix!Last week, Netflix welcomed about 150 women to its campus for a set of lightning talks followed by demos and networking. Organized in coordination with CloudNOW, the event was high on the fun and high on the tech. Lightning talks included: Devika Chawla, Director of Engineering, Netflix. Devika’s talk was…Cloudnow3 min read
Published in Netflix TechBlog·Mar 13, 2014NetflixOSS Season 2, Episode 1by Ruslan Meshenberg Wondering what this headline means? It means that NetflixOSS continues to grow, both in the number of projects that are now available and in the use by others. We held another NetflixOSS Meetup in our Los Gatos, Calif., headquarters last night. …Netflixoss4 min read
Published in Netflix TechBlog·Mar 3, 2014The Netflix Dynamic Scripting PlatformThe Engine that powers the Netflix API — by Sangeeta Narayanan Over the past couple of years, we have optimized the Netflix API with a view towards improving performance and increasing agility. …API7 min read
Published in Netflix TechBlog·Feb 27, 2014Netflix Hack Dayby Daniel Jacobson, Ruslan Meshenberg, Matt McCarthy and Leslie Posada At Netflix, we pride ourselves in creating a culture of innovation and experimentation. We are constantly running A/B tests on virtually every enhancement to the Netflix experience. There are other ways in which we instill this culture within Netflix, including…Hackathons3 min read
Published in Netflix TechBlog·Feb 10, 2014Distributed Neural Networks with GPUs in the AWS Cloudby Alex Chen, Justin Basilico, and Xavier Amatriain As we have described previously on this blog, at Netflix we are constantly innovating by looking for better ways to find the best movies and TV shows for our members. When a new algorithmic technique such as Deep Learning shows promising results…AWS11 min read
Published in Netflix TechBlog·Jan 16, 2014Improving Netflix’s Operational Visibility with Real-Time Insight Toolsby Ranjit Mavinkurve, Justin Becker, and Ben Christensen For Netflix to be successful, we have to be vigilant in supporting the tens of millions of connected devices that are used by our 40+ million members throughout 40+ countries. …Data Visualization7 min read
Published in Netflix TechBlog·Jan 9, 2014S3mper: Consistency in the Cloudby Daniel C. Weeks In previous posts, we discussed how the Hadoop platform at Netflix leverages AWS’s S3 offering (read more here). In short, Netflix considers S3 the “source of truth” for all data warehousing. There are many attractive features that draw us to this service including: 99.999999999% durability, 99.99%…Big Data9 min read
Published in Netflix TechBlog·Jan 2, 2014Introducing PigPen: Map-Reduce for Clojureby Matt Bossenbroek It is our pleasure to release PigPen to the world today. PigPen is map-reduce for Clojure. It compiles to Apache Pig, but you don’t need to know much about Pig to use it. What is PigPen? A map-reduce language that looks and behaves like clojure.core The ability to write map-reduce…Clojure17 min read
Published in Netflix TechBlog·Dec 20, 2013Astyanax Updateby Puneet Oberai & Christos Kalantzis Overview Astyanax is one of Netflix’s more popular OSS Projects. …Cassandra12 min read
Published in Netflix TechBlog·Dec 19, 2013Netflix Presentation Videos from AWS Re:Invent 2013AWS recorded all the talks, there are hundreds of videos, so to make it easier to find the Netflix-related ones, here are links to the videos. DMG206 — Development Patterns for Iteration, Scale, Performance and Availability Neil Hunt — Chief Product Officer ENT203 What Enterprises Can Learn From Netflix…Availability1 min read
Published in Netflix TechBlog·Dec 16, 2013Pioneering application design on TVs & TV-connected devicesNetflix recently launched the latest evolution of our core app for an increasing number of TVs and TV-connected devices. The app represents a unique mixture of platform and user interface design innovations. …Accelerated Compositing4 min read
Published in Netflix TechBlog·Dec 11, 2013Aegisthus is Now Part of NetflixOSSby Charles Smith and Jeff Magnusson Today we are happy to announce that Aegisthus, our map/reduce program for reading Cassandra SSTables, is open source. Aegisthus is the tool we use daily to make the data stored in Cassandra available for analysis on our big data platform. …Aegisthus6 min read
Published in Netflix TechBlog·Dec 9, 2013Announcing Suro: Backbone of Netflix’s Data Pipelineby Jae Hyeon Bae, Danny Yuan, and Sudhir Tonse To make the best business and technical decisions, it is critical for Netflix to reliably collect application specific data in a timely fashion. At Netflix we deploy a fairly large number of AWS EC2 instances that host our web services and…Cloud7 min read
Published in Netflix TechBlog·Dec 5, 2013Announcing Zeno — Netflix’s In-Memory Data Distribution Frameworkby Drew Koszewnik Netflix’s Video Metadata Service (VMS) is the platform that supplies all of the data about our movies and TV shows necessary to drive the Netflix experience. This data ranges from video titles and synopses to the resolutions and bitrates of streams for playback. In previous blog posts…Netflixoss10 min read
Published in Netflix TechBlog·Dec 4, 2013Scryer: Netflix’s Predictive Auto Scaling Engine — Part 2by Danny Yuan, Neeraj Joshi, Daniel Jacobson, Puneet Oberai In part 1 of this series, we introduced Scryer, Netflix’s predictive autoscaling engine, and discussed its use cases and how it runs in Netflix. …Analytics9 min read
Published in Netflix TechBlog·Nov 18, 2013Preparing the Netflix API for DeploymentAt Netflix, we are committed to bringing new features and product enhancements to customers rapidly and frequently. Consequently, dozens of engineering teams are constantly innovating on their services, resulting in a rate of change to the overall service is vast and unending. …Continuous Delivery8 min read
Published in Netflix TechBlog·Nov 18, 2013Building the New Netflix Experience for TVby Joubert Nel We just launched a new Netflix experience for TV and game consoles. The new design is based on our premise that each show or movie has a tone and a narrative that should be conveyed by the UI. To tell a richer story we provide relevant evidence…Device5 min read
Published in Netflix TechBlog·Nov 14, 2013Netflix Open Source Software Cloud Prize WinnersWe launched the Netflix Open Source Software Cloud Prize in March 2013 and it got a lot of attention in the press and blogosphere. Six months later we closed the contest, took a good look at the entrants, picked the best as nominees, and the panel of distinguished judges decided…Cloud Prize8 min read
Published in Netflix TechBlog·Nov 5, 2013Scryer: Netflix’s Predictive Auto Scaling Engineby Daniel Jacobson, Danny Yuan, and Neeraj Joshi To deliver the best possible experience to Netflix customers around the world, it is critical for us to maintain a robust, scalable, and resilient system. That is why we have built (and open sourced) applications ranging from Hystrix to Chaos Monkey. All…Analytics5 min read
Published in Netflix TechBlog·Oct 21, 2013HTML5 Video Playback UIby Kristofer Baxter In the past we’ve written about HTML5 Video (HTML5 Video in IE11 on Windows 8.1, and HTML5 Video at Netflix) but we haven’t spoken much about how we built the player UI. …Html 56 min read
Published in Netflix TechBlog·Oct 10, 2013Introducing Chaos to C*by Christos Kalantzis, Minh Do, Homajeet Cheema, and Roman Vasilyev One of the practices that sets Netflix apart from most companies is the belief that you can only know how good your software stack is by trying to making it fail. We’ve blogged about Chaos Monkey and how it helps…Cassandra2 min read
Published in Netflix TechBlog·Sep 26, 2013NetflixOSS Meetup S1E4 — Cloud Prize Nominationsby Adrian Cockcroft We launched the Netflix Open Source Software Cloud Prize in March 2013 and it got a lot of attention in the press and blogosphere. Six months later we closed the contest, took a good look at the entrants, picked the best as nominees and announced them at…Cloud Prize9 min read
Published in Netflix TechBlog·Aug 14, 2013Deploying the Netflix APIby Ben Schmaus As described in previous posts (“Embracing the Differences” and “Optimizing the Netflix API”), the Netflix API serves as an integration hub that connects our device UIs to a distributed network of data services. In supporting this ecosystem, the API needs to integrate an ever-evolving set of features…Continuous Delivery10 min read
Published in Netflix TechBlog·Jul 25, 2013NetflixOSS Meetup Series 1 Episode 3 — Featured Contributionsby Adrian Cockcroft and Ruslan Meshenberg Our third NetflixOSS Meetup introduced our latest project releases, updates on the NetflixOSS Cloud Prize, and featured demonstrations from contributors as well as many Netflix projects. The projects we covered in the lightning presentations were: Pytheas — A web based framework for quickly building…Cloud Computing3 min read
Published in Netflix TechBlog·Jun 26, 2013HTML5 Video in IE 11 on Windows 8.1by Anthony Park and Mark Watson. We’ve previously discussed our plans to use HTML5 video with the proposed “Premium Video Extensions” in any browser which implements them. …Html 52 min read
Published in Netflix TechBlog·Jun 26, 2013Introducing Lipstick on A(pache) Pigby Jeff Magnusson, Charles Smith, John Lee, and Nathan Bates We’re pleased to announce Lipstick (our Pig workflow visualization tool) as the latest addition to the suite of Netflix Open Source Software. At Netflix, Apache Pig is used heavily amongst developers when productionizing complex data transformations and workflows against…Big Data3 min read
Published in Netflix TechBlog·Jun 21, 2013Genie is out of the bottle!by Sriram Krishnan In a prior tech blog, we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazon’s Simple Storage Service (S3) as our “source of truth”, leveraging the elasticity of the cloud to run multiple dynamically…Big Data9 min read
Published in Netflix TechBlog·Jun 14, 2013Isthmus — Resiliency against ELB outagesOn Christmas Eve, 2012, Netflix streaming service experienced an outage. For full details, see “A Closer Look at the Christmas Eve Outage” by Adrian Cockcroft: A Closer Look at the Christmas Eve Outage what happened and how we continue to invest in higher availability solutionsmedium.com This outage was particularly painful, both because of the timing, as well as the root cause — ELB control plane, was outside of our ability…AWS5 min read
Published in Netflix TechBlog·Jun 12, 2013Women in Cloud Meetup at Netflixby Shobana Radhakrishnan We recently held a Meetup on our campus for Bay Area women in the Cloud Space, in collaboration with Cloud-NOW. Women from across a number of companies and backgrounds related to Cloud attended the event and participated in talks and panel discussions on various technical topics related…Cloud Computing4 min read
Published in Netflix TechBlog·Jun 12, 2013Announcing Zuul: Edge Service in the CloudThe Netflix streaming application is a complex array of intertwined systems that work together to seamlessly provide our customers a great experience. The Netflix API is the front door to that system, supporting over 1,000 different device types and handing over 50,000 requests per second during peak hours. We are…API7 min read
Published in Netflix TechBlog·May 29, 2013Conformity Monkey — Keeping your cloud instances following best practicesby Michael Fu and Cory Bennett, Engineering Tools Cloud computing makes it much easier to launch new applications or start new instances. At Netflix, engineers can easily launch a new application in Asgard with a few clicks. With this freedom there are sometimes consequences where launched applications or instances may…AWS4 min read
Published in Netflix TechBlog·May 8, 2013Object Cache for Scaling Video Metadata Managementby Shobana Radhakrishnan As the largest Internet TV network, one of the most interesting challenges we face at Netflix is scaling services to the ever-increasing demands of over 36 million customers from over 40 countries. Each movie or TV show on Netflix is described by a complex set of metadata…Scaling6 min read
Published in Netflix TechBlog·Apr 15, 2013HTML5 Video at Netflixby Anthony Park and Mark Watson Today, we’re excited to talk about proposed extensions to HTML5 video that enable playback of premium video content on the web. We currently use Microsoft Silverlight to deliver streaming video to web browsers on the PC and Mac. It provides a high-quality streaming experience…Video4 min read
Published in Netflix TechBlog·Mar 29, 2013NetflixOSS Meetup Series 1 Episode 2 — Announcing the Cloud Prizeby Adrian Cockcroft On March 13th we held our second NetflixOSS meetup. It was well attended and used the same format of some presentations followed by demonstrations of the latest projects. Videos and slides are embedded below. We started off with lightning talks by the engineers who own Karyon, our…Animator4 min read
Published in Netflix TechBlog·Mar 27, 2013System Architectures for Personalization and Recommendationby Xavier Amatriain and Justin Basilico In our previous posts about Netflix personalization, we highlighted the importance of using both data and algorithms to create the best possible experience for Netflix members. We also talked about the importance of enriching the interaction and engaging the user with the recommendation system…Cloud Architecture11 min read
Published in Netflix TechBlog·Mar 22, 2013AMI Creation with Aminatorby Michael Tripoli & Karate Vick Aminator is a tool for creating custom Amazon Machine Images (AMIs). It is the latest implementation of a series of AMI creation tools that we have developed over the past three years. …AWS6 min read
Published in Netflix TechBlog·Mar 12, 2013Public Continuous Integration Builds for our OSS Projectsby Gareth Bowles As Netflix continues to open source our platform (26 projects at the last count), our GitHub repos are seeing more and more changes, both commits from the project owners and pull requests from external contributors. To provide more visibility into the quality of those changes, we’re setting…Jenkins3 min read
Published in Netflix TechBlog·Mar 11, 2013Introducing the first NetflixOSS Recipe: RSS Readerby Prasanna Padmanabhan, Shashi Madappa, Kedar Sadekar, and Chris Fregly Over the past year, Netflix has open sourced many of its components such as Hystrix, Eureka, Servo, Astyanax, Ribbon, etc. While we continue to open source more of our components, it would be useful to open source a set of…Archaius4 min read
Published in Netflix TechBlog·Mar 11, 2013Python at Netflixby Roy Rapoport, Brian Moyles, Jim Cistaro, and Corey Bertram We’ve blogged a lot about how we use Java here at Netflix, but Python’s footprint in our environment continues to increase. In honor of our sponsorship of PyCon, we wanted to highlight our many uses of Python at Netflix. Developers…Python6 min read
Published in Netflix TechBlog·Feb 25, 2013Announcing EVCache: Distributed in-memory datastore for Cloudby Shashi Madappa EVCache is a distributed in-memory caching solution based on memcached & spymemcached that is well integrated with Netflix OSS and AWS EC2 infrastructure. Today we are announcing the open sourcing of EVCache client library on Github. EVCache is an abbreviation for: Ephemeral — The data stored is…Cloud Computing5 min read
Published in Netflix TechBlog·Feb 14, 2013Netflix Queue: Data migration for a high volume web applicationby Prasanna Padmanabhan & Shashi Madappa There will come a time in the life of most systems serving data, when there is a need to migrate data to a more reliable, scalable and high performance data store while maintaining or improving data consistency, latency and efficiency. This document explains the…Cassandra8 min read
Published in Netflix TechBlog·Feb 12, 2013Active-Active for Multi-Regional Resiliencyby Ruslan Meshenberg, Naresh Gopalani, and Luke Kosewski In June, we talked about Isthmus — our approach to achieve resiliency against region-wide ELB outage. After completing the Isthmus project we embarked on the next step in our quest for higher resiliency and availability — a full multi-regional Active-Active solution. …Cloud Computing12 min read
Published in Netflix TechBlog·Feb 11, 2013First NetflixOSS Meetupby Adrian Cockcroft and Ruslan Meshenberg The inaugural Netflix Open Source Software (NetflixOSS) meetup was held at Netflix headquarters in Los Gatos California on February 6th 2013. We had over 200 attendees; individual developers, academics, vendors and companies, from small startups to the largest global corporations. For those who missed…Open Source Software7 min read
Published in Netflix TechBlog·Feb 4, 2013Reactive Programming in the Netflix API with RxJavaby Ben Christensen and Jafar Husain Our recent post on optimizing the Netflix API introduced how our web service endpoints are implemented using a reactive programming model for composition of asynchronous callbacks from our service layer. Optimizing the Netflix API how we redesigned our API to help UI teams meet their needsmedium.com This post takes a closer look at how and why we use the reactive…Rxjava7 min read
Published in Netflix TechBlog·Jan 28, 2013Announcing Ribbon: Tying the Netflix Mid-Tier Services Togetherby Allen Wang and Sudhir Tonse Netflix embraces a fine-grained Service Oriented Architecture as the underpinning of its Cloud based deployment model. Currently, we run hundreds of such fine grained services that are collectively responsible in handling the customer facing requests via a few “Edge Services” such as the Netflix…Cloud Architecture5 min read
Published in Netflix TechBlog·Jan 18, 2013NetflixGraph Metadata Library: An Optimization Case Studyby Drew Koszewnik Here at Netflix, we serve more than 30 million subscribers across over 40 countries. These users collectively generate billions of requests per day, most of which require metadata about our videos. Each call we receive can potentially cull through thousands of video metadata attributes. …Programming8 min read
Published in Netflix TechBlog·Jan 16, 2013Reactive Programming at NetflixHi, my name is Jafar Husain. I’m a Senior Developer on the TV User Interface team. Over the last year, Netflix has reinvented our client-server interaction model. One of the key building blocks of our platform is Microsoft’s open-source Reactive Extensions library (Rx). …Java Script3 min read
Published in Netflix TechBlog·Jan 15, 2013Optimizing the Netflix APIby Ben Christensen About a year ago the Netflix API team began redesigning the API to improve performance and enable UI engineering teams within Netflix to optimize client applications for specific devices. …API6 min read
Published in Netflix TechBlog·Jan 14, 2013NetflixOSS Open Houseby Adrian Cockcroft, Ruslan Meshenberg. As many of you know, we have open sourced many components of the Netflix infrastructure over the last year — ranging from our deployment console Asgard and Simian Army Monkeys, to the Cassandra and Zookeeper clients that we use internally. We hope that these tools…Open Source2 min read
Published in Netflix TechBlog·Jan 10, 2013Hadoop Platform as a Service in the Cloudby Sriram Krishnan and Eva Tse, Cloud Big Data Platform Hadoop has become the de facto standard for managing and processing hundreds of terabytes to petabytes of data. At Netflix, our Hadoop-based data warehouse is petabyte-scale, and growing rapidly. However, with the big data explosion in recent times, even this…Big Data11 min read
Published in Netflix TechBlog·Jan 4, 2013Janitor Monkey — Keeping the Cloud Tidy and Cleanby Michael Fu and Cory Bennett, Engineering Tools One of the great advantages of moving from a private datacenter into the cloud is that you have quick and easy access to nearly limitless new resources. Innovation and experimentation friction is greatly reduced: to push out a new application release you…AWS6 min read
Published in Netflix TechBlog·Dec 31, 2012A Closer Look at the Christmas Eve Outageby Adrian Cockcroft Netflix streaming was impacted on Christmas Eve 2012 by problems in the Amazon Web Services (AWS) Elastic Load Balancer (ELB) service that routes network traffic to the Netflix services supporting streaming. The postmortem report by AWS can be read here. Summary of the December 24, 2012 Amazon ELB Service Event in the US-East Region We would like to share more details with our customers about the event that occurred with the Amazon Elastic Load…amzn.toAWS5 min read
Published in Netflix TechBlog·Dec 17, 2012Complexity In The Digital Supply ChainNetflix launched in Denmark, Norway, Sweden, and Finland on Oct. 15th. I just returned from a trip to Europe to review the content deliveries with European studios that prepared content for this launch. This trip reinforced for me that today’s Digital Supply Chain for the streaming video industry is awash…Technology6 min read
Published in Netflix TechBlog·Dec 11, 2012Hystrix Dashboard + Turbine Stream Aggregatorby Ben Christensen, Puneet Oberai and Ben Schmaus Two weeks ago we introduced Hystrix, a library for engineering resilience into distributed systems. Today we’re open sourcing the Hystrix dashboard application, as well as a new companion project called Turbine that provides low latency event stream aggregation. The Hystrix dashboard has…API3 min read
Published in Netflix TechBlog·Dec 10, 2012Videos of the Netflix talks at AWS Re:Inventby Adrian Cockcroft Most of the talks and panel sessions at AWS Re:Invent were recorded, but there are so many sessions that it’s hard to find the Netflix ones. Here’s a link to all of the videos posted by AWS that mention Netflix: http://www.youtube.com/user/AmazonWebServices/videos?query=netflix They are presented below in what…AWS3 min read
Published in Netflix TechBlog·Dec 3, 2012AWS Re:Invent was Awesome!by Adrian Cockcroft There was a very strong Netflix presence at AWS Re:Invent in Las Vegas this week, from Reed Hastings appearing in the opening keynote, to a packed series of ten talks by Netflix management and engineers, and our very own expo booth. …AWS6 min read
Published in Netflix TechBlog·Nov 26, 2012Introducing Hystrix for Resilience Engineeringby Ben Christensen In a distributed environment, failure of any given service is inevitable. Hystrix is a library designed to control the interactions between these distributed services providing greater tolerance of latency and failure. Hystrix does this by isolating points of access between the services, stopping cascading failures across them…API2 min read
Published in Netflix TechBlog·Nov 20, 2012Announcing Blitz4j — a scalable logging frameworkby Karthikeyan Ranganathan We are proud to announce Blitz4j , a critical component of the Netflix logging infrastructure that helps Netflix achieve high volume logging without affecting scalability of the applications. What is Blitz4j? Blitz4j is a logging framework built on top of log4j to reduce multithreaded contention and enable highly scalable logging…Appender6 min read
Published in Netflix TechBlog·Nov 6, 2012Edda — Learn the Stories of Your Cloud Deploymentsby Cory Bennett, Engineering Tools Operating “in the cloud” has its challenges, and one of those challenges is that nothing is static. Virtual host instances are constantly coming and going, IP addresses can get reused by different applications, and firewalls suddenly appear as security configurations are updated. At Netflix we…Cloud Computing4 min read
Published in Netflix TechBlog·Oct 29, 2012Post-mortem of October 22, 2012 AWS degradationby Jeremy Edberg & Ariel Tseitlin On Monday, October 22nd, Amazon experienced a service degradation. It was highly publicized due to the fact that it took down many popular websites for multiple hours. Netflix however, while not completely unscathed, handled the outage with very little customer impact. …AWS5 min read
Published in Netflix TechBlog·Oct 29, 2012Governator — Lifecycle and Dependency Injectionby Jordan Zimmerman Yahhh — PostConstruct me now unt PreDestroy me lay-tuh Governator Governator is a library of extensions and utilities that extend Google Guice to provide: Classpath scanning and automatic binding Lifecycle management Configuration to field mapping Field validation Parallelized object warmup Lazy singleton support Generic binding annotations Governator is…AWS2 min read
Published in Netflix TechBlog·Sep 6, 2012Netflix @ Recsys 2012by Xavier Amatriain We are just a few days away from the 2012 ACM Recommender Systems Conference (#Recsys2012), that this year will take place in Dublin, Ireland. …Recommendations2 min read
Published in Netflix TechBlog·Sep 4, 2012Netflix Shares Cloud Load Balancing And Failover Tool: Eureka!by Karthikeyan Ranganathan We are proud to announce Eureka, a service registry that is a critical component of the Netflix infrastructure in the AWS cloud, and underpins our mid-tier load balancing, deployment automation, data storage, caching and various other services. Netflix/eureka eureka - AWS Service registry for resilient mid-tier load balancing and failover.github.com What is Eureka? Eureka is a REST based service that is primarily used…Eureka6 min read
Published in Netflix TechBlog·Jul 19, 2012Benchmarking High Performance I/O with SSD for Cassandra on AWSby Adrian Cockcroft Today AWS has launched a new Solid State Disk (SSD) based instance that addresses the need for high performance I/O, and we have run a few initial benchmarks to see how it shapes up. With this announcement AWS makes it easy to provision extremely high I/O capacity…Cloud Computing9 min read
Published in Netflix TechBlog·Jul 18, 2012Enabling Support for IPv6by Rajiv Aggarwal and David Temkin In our first post about World IPv6 Launch we talked about our motivations for supporting IPv6. Now we’ll dive into some of the technical challenges we faced and how we addressed them. Our primary concern was ensuring uptime and quality of experience for our…Ipv 64 min read
Published in Netflix TechBlog·Jul 13, 2012Open Source at Netflixby Ruslan Meshenberg At Netflix we use a wide range of Open Source technologies. In the recent months, we also released many of our internally developed components and libraries, starting with Curator for Zookeeper, and most recently with Asgard. We started down this path by becoming a big user of…Cloud Computing3 min read
Published in Netflix TechBlog·Jul 9, 2012Embracing the Differences : Inside the Netflix API RedesignAs I discussed in my recent blog post on ProgrammableWeb.com, Netflix has found substantial limitations in the traditional one-size-fits-all (OSFA) REST API approach. As a result, we have moved to a new, fully customizable API. The basis for our decision is that Netflix’s streaming service is available on more than…API9 min read
Published in Netflix TechBlog·Jun 28, 2012Scalable Logging and Trackingby Kedar Sadekar At Netflix we work hard to improve personalized recommendations. We use a lot of data to make recommendations better. What may seem an arbitrary action — scrolling up, down, left or right and how much — actually provides us with valuable information. …AWS7 min read
Published in Netflix TechBlog·Jun 25, 2012Asgard: Web-based Cloud Management and Deploymentby Joe Sondow, Engineering Tools For the past several years Netflix developers have been using self-service tools to build and deploy hundreds of applications and services to the Amazon cloud. One of those tools is Asgard, a web interface for application deployments and cloud management. Asgard is named for the…AWS9 min read
Published in Netflix TechBlog·Jun 20, 2012Netflix Recommendations: Beyond the 5 stars (Part 2)by Xavier Amatriain and Justin Basilico (Personalization Science and Engineering) In part one of this blog post, we detailed the different components of Netflix personalization. We also explained how Netflix personalization, and the service as a whole, have changed from the time we announced the Netflix Prize. Netflix Recommendations: Beyond the 5 stars (Part 1) One of the most valued Netflix assets is our recommendation systemmedium.comData Science10 min read
Published in Netflix TechBlog·Jun 18, 2012Announcing Archaius: Dynamic Properties in the Cloudby Allen Wang and Sudhir Tonse Netflix has a culture of being dynamic when it comes to decision making. This trait comes across both in the business domain as well as in technology and operations. It follows that we like the ability to effect changes in the behavior of our…Cloud Architecture7 min read
Published in Netflix TechBlog·Apr 6, 2012Netflix Recommendations: Beyond the 5 stars (Part 1)by Xavier Amatriain and Justin Basilico (Personalization Science and Engineering) In this two-part blog post, we will open the doors of one of the most valued Netflix assets: our recommendation system. In Part 1, we will relate the Netflix Prize to the broader recommendation challenge, outline the external components of…Machine Learning9 min read
Published in Netflix TechBlog·Mar 14, 2012Testing Netflix on AndroidWhen Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: a) We wanted to release rapidly every 6–8 weeks, b) There were hundreds of Android devices of different shapes, versions, capacities and specifications which need to playback audio and video and c) We wanted to…Android App Development4 min read
Published in Netflix TechBlog·Mar 13, 2012JMeter Plugin for Cassandraby Vijay Parthasarathy and Denis Sheahan A number of previous blogs have discussed our adoption of Cassandra as a NoSQL solution in the cloud. We now have over 55 Cassandra clusters in the cloud and are moving our source of truth from our Datacenter to these Cassandra clusters. As part…Cassandra5 min read
Published in Netflix TechBlog·Feb 29, 2012Fault Tolerance in a High Volume, Distributed Systemby Ben Christensen In an earlier post by Ben Schmaus, we shared the principles behind our circuit-breaker implementation. …API8 min read
Published in Netflix TechBlog·Feb 27, 2012Aegisthus — A Bulk Data Pipeline out of Cassandraby Charles Smith and Jeff Magnusson Our job in Data Science and Engineering is to consume all the data that Netflix produces and provide an offline batch-processing platform for analyzing and enriching it. As has been mentioned in previous posts, Netflix has recently been engaged in making the transition to…Aegisthus5 min read
Published in Netflix TechBlog·Feb 21, 2012Announcing Priamby Praveen Sadhu, Vijay Parthasarathy & Aditya Jami We talked in the past about our move to NoSQL and Cassandra has been a big part of that strategy. Cassandra hit a big milestone recently with the announcement of the v1 release. …Cassandra7 min read
Published in Netflix TechBlog·Feb 10, 2012Announcing Servoby Brian Harrington & Greg Orzell In a previous blog post about auto scaling, I mentioned that we would be open sourcing the library that we use to expose application metrics. Servo is that library. It is designed to make it easy for developers to export metrics from their application…AWS4 min read
Published in Netflix TechBlog·Jan 27, 2012Ephemeral Volatile Caching in the cloudby Shashi Madappa In most applications there is some amount of data that will be frequently used. Some of this data is transient and can be recalculated, while other data will need to be fetched from the database or a middle tier service. In the Netflix cloud architecture we use…Cloud Architecture9 min read
Published in Netflix TechBlog·Dec 8, 2011Making the Netflix API More Resilientby Ben Schmaus The API brokers catalog and subscriber metadata between internal services and Netflix applications on hundreds of device types. If any of these internal services fail there is a risk that the failure could propagate to the API and break the user experience for members. To provide the…API7 min read
Published in Netflix TechBlog·Nov 29, 2011Introducing Curator — The Netflix ZooKeeper Libraryby Jordan Zimmerman Open Source at Netflix We are committed to Open Source Software at Netflix. We’ve blogged about it in the past. Today we are announcing a portal for open source projects from Netflix. The portal is currently hosted on Github. …Cloud4 min read
Published in Netflix TechBlog·Nov 2, 2011Benchmarking Cassandra Scalability on AWS — Over a million writes per secondby Adrian Cockcroft and Denis Sheahan Netflix has been rolling out the Apache Cassandra NoSQL data store for production use over the last six months. As part of our benchmarking we recently decided to run a test designed to validate our tooling and automation scalability as well as the performance…AWS13 min read
Published in Netflix TechBlog·Oct 12, 2011Netflix Performance on Top NetworksHi, it’s Ken Florance, Director of Content Delivery for Netflix, with an updated “Netflix Performance on Top Networks” chart. As promised, we’ve been able to separate AT&T and Verizon FTTx offerings from their DSL offerings. The chart now gives a fairly complete look at performance on top networks, with additional insight into how different technologies (DSL, Cable, FTTx) impact potential throughput.Netflix1 min read
Published in Netflix TechBlog·Aug 13, 2011Building with LegosIn the six years that I have been involved in building and releasing software here at Netflix, the process has evolved and improved significantly. When I started, we would build a WAR, get it setup and tested on a production host, and then run a script that would stop tomcat…Build3 min read
Published in Netflix TechBlog·Jul 19, 2011The Netflix Simian ArmyWe’ve talked a bit in the past about our move to the cloud, and John shared some of our lessons learned in going through that transition in a previous post. …Chaos Monkey4 min read
Published in Netflix TechBlog·Jun 17, 2011Upcoming Changes to the Open API ProgramThis is Daniel Jacobson, director of engineering, with a message to everyone in the Netflix Open API community. We’re making some changes to the Open API program to support the Netflix focus on international streaming. Later this year, we will discontinue support for DVD-related features, degrading them through redirects or…API1 min read
Published in Netflix TechBlog·May 31, 2011Netflix Performance on Top ISP NetworksHello all, it’s Ken Florance, Director of Content Delivery for Netflix. As you can see, we’ve updated our “Netflix Performance on Top Networks” chart showing performance on Top ISPs over the last several months. We first published similar data in January and we were pleased with the feedback we received…Isps2 min read
Published in Netflix TechBlog·Apr 29, 2011Lessons Netflix Learned from the AWS OutageOn Thursday, April 21st, Amazon experienced a large outage in AWS US-East which they describe here. This outage was highly publicized because it took down or severely hampered a number of popular websites that depend on AWS for hosting. …Cloud Computing8 min read
Published in Netflix TechBlog·Mar 29, 2011NoSQL @ Netflix Talk (Part 1)A few weeks ago, I gave the first in a series of planned talks on the topic of NoSQL @ Netflix. By now, it is widely known that Netflix has achieved something remarkable over the past 2 years — accelerated subscriber growth with an ever-improving streaming experience. …No SQL2 min read
Published in Netflix TechBlog·Mar 8, 2011Cloud Connect Keynote: Complexity and FreedomNetflix In The Clouds — View more presentations from Kevin McEntee On March 8th, 2011, I was fortunate to be able to deliver 10 minutes of the keynote address for the Cloud Connect conference in Santa Clara, California. Here are some of the points I made during the talk. Availability We started this cloud re-architecture…Cloud Computing5 min read
Published in Netflix TechBlog·Feb 8, 2011Redesigning the Netflix APIThis is Daniel Jacobson, Director of Engineering for the API here at Netflix. The Netflix API launched in 2008 with a focus on the public developer community. The expectation was that this community would build amazing and inspiring applications that would take Netflix to a new level in serving our…API4 min read
Published in Netflix TechBlog·Jan 28, 2011NoSQL at NetflixThis is Yury Izrailevsky, Director of Cloud and Systems Infrastructure here at Netflix. As Netflix moved into the cloud, we needed to find the appropriate mechanisms to persist and query data within our highly distributed infrastructure. Our goal is to build fast, fault tolerant systems at Internet scale. We realized…Cassandra4 min read
Published in Netflix TechBlog·Jan 27, 2011Netflix Performance on Top ISP NetworksHi there. This is Ken Florance, Director of Content Delivery here at Netflix. As we continue to stream more and more great movies and TV shows, we find ourselves in the unique position of having insight into the performance of hundreds of millions of long duration, high-definition video streams delivered…Netflix2 min read
Published in Netflix TechBlog·Jan 19, 2011How We Determine Product SuccessAt Netflix we engage in what we call consumer science: we test new ideas with real customers, at scale, and we measure for statistically significant differences in how they engage with our product. Are members staying with the service longer? …Product Management5 min read
Published in Netflix TechBlog·Dec 22, 2010HTML5 and Video StreamingThis is Christian Kaiser, VP of Engineering here at Netflix. My colleague John Ciancutti recently posted about our use of HTML5 for user interfaces; I thought I’d add a perspective on HTML5 and video streaming. Since HTML5 includes a facility to embed video playback (the <video> tag), it seems like…Video3 min read
Published in Netflix TechBlog·Dec 16, 20105 Lessons We’ve Learned Using AWSIn my last post I talked about some of the reasons we chose AWS as our computing platform. We’re about one year into our transition to AWS from our own data centers. We’ve learned a lot so far, and I thought it might be helpful to share with you some…Cloud Computing4 min read
Published in Netflix TechBlog·Dec 14, 2010Four Reasons We Choose Amazon’s Cloud as Our Computing PlatformOne year ago, none of Netflix’s customer traffic was supported out of AWS, Amazon’s cloud services. Today, most of our member traffic is supported by software we’ve built and deployed in AWS. This includes much of our member website at Netflix.com …Cloud Computing3 min read
Published in Netflix TechBlog·Dec 10, 2010Why we use and contribute to open source softwareThis is Kevin McEntee, VP of Systems & ECommerce Engineering here at Netflix. Netflix is a technology company. We develop and apply great software technology to deliver a great streaming video experience. Our budget, measured in dollars, time, people, and energy, is limited and we must therefore focus our technology…Open Source2 min read
Published in Netflix TechBlog·Dec 1, 2010Netflix Tech BlogHi there, This is a new Netflix blog focused purely on technology issues. We’ll share our perspectives, decisions and challenges regarding the software we build and use to create the Netflix service. Netflix is a software company. We don’t sell software, but nevertheless software is our lifeblood. We’ve been a…Netflix1 min read
